texts: Numpy arithmetic operation between two columns

texts:
Below is the numpy array. I need to perform two operations on this array.

Add one column with value [column 1] - [column 3].
Add another column with value [column 1] - [previous value of column1].

I can do this using normal list operations, but is it possible to use numpy or pandas? If so, how can it be done?

Input data:

[['78' '3412' '98' '3441']
 ['106' '3412' '127' '3434']
 ['139' '3411' '160' '3434']
 ['170' '3411' '191' '3442']
]


texts: Are there more layer types like convolution layers and fully connected layers?

texts:
I've been learning about neural networks and I'm curious: Are there any other layer types like convolution and fully connected layers? I know there are things like max pooling or various regularization layers, but I'm mainly interested in more variations on fully connected layers (like how convolution layers do the same thing as fully connected layers but take the shape of data into account). It just seems strange that we wouldn't need anything except straight matrix multiplication or matrix multiplication on smaller chunks.

Do these exist? If not, why not?


texts:
I am doing simple CNN based binary image classification on classifying types of waste as organic and recyclable using image augmentation. How ever I have a doubt.

In the third figure i have set the steps_per_epoch = 22564/32= 705 where 32 is batch size and validation_steps as 2513/32 = 79 as per documentation does that mean out of 22564 training and 2513 validation samples 705 and 79 are trained and validated ?


texts: global flight network (with number of pessengers per year) dataset

texts:
Is there such a global flight network dataset that gives you:

flight route (the connection between each airports or cities by some flight company)
the number of passengers (per year) taking each route.

In other words the dataset is a directed flight graph with passenger volume as the edge weight.

The closest thing I can find is OpenFlights route data.


texts: Adding recommendations to the output of a classification model

texts:
I have built a binary classification model using:

logit
decision trees
random forest
bagging classifier
gradientboost
xgboost
adaboost

I have evaluated the above models and chose xgboost based on training/test and validation metrics (accuracy, prediction, recall, f1 and AUC).

I want to now productionalize it and share the output with the business. The output would basically have a list of items with the predicted class and that could be filtered based on business needs.

However, Instead of simply giving the business the predicted classes, I want to add insights/recommendations as to why a specific item was predicted with class X and how you could go about working on the item to change its class from say X to Y.

How do I go about this? I thought of using feature importance, but my input data shape is [800,000 * 1,050] and I am not sure if it would the best way to proceed.

Are there any existing industry standard methodologies that can add interpretability to such models and convert them from a black box models to prescriptive models?


texts: How do I identify clusters that match on categorical data?

texts:
I am seeking some directions for a proper path to research the solve for this problem:

My company made all our employees take a "StrengthFinders" test, which results in every employee being assigned their top five (ordered) "strengths" from a possible list of 34 strengths.  We have 500 employees.  I am supposed to identify all the employees that match each other for the same 5 strengths (order not important), and also for employees that match each other for 4 out of 5 strengths (again, order doesn't matter).  I could potentially have multiple groups matching on different sets of strengths, e.g.:
Group 1: Billy, Sally, Michael have strengths A, H, I, K, Z
Group 2: Bobby and Suzy have strengths A, B, L, S, W

For the case where strengths match for 4 out of 5, I might have the same people from Group 1 above, plus Joe, whose strengths are A, H, M, K, Z; and
Seth, whose strengths are A, H, G, K, Z.  I would expect more groupings for the case of 4 out of 5 than the 5 out of 5 case.

The strengths are categorical in nature, so what I've read so far has largely revolved around clustering of continuous numerical variables.

I am looking for an algorithmic way to identify clusters and the members of those clusters for this situation.  I think I could do this brute force by repeatedly sorting data in Excel, but I'm confident that a better way must exist, and I ask you to point me in that direction.  Thank you.


texts:
I am trying to build model to convert Sign Language to text. I am facing some problem while trying to create and using tensorboard object inorder to visually see the output and optimize my model.

from tensorflow.keras.layers import Dense, Activation, Conv2D, Flatten, MaxPooling2D,Dropout
from tensorflow.keras.models import Sequential
from tensorflow.keras.utils import normalize
from tensorflow.keras.callbacks import TensorBoard
import numpy as np
import tensorflow as tf
import pickle
import cv2
import time
# Load the dataset

X = pickle.load(open("X_ab.pickle", "rb"))
Y = pickle.load(open("Y_ab.pickle", "rb"))

modelname = "a-z{}".format(int(time.time()))
board = TensorBoard(Log_dir="logs/{}".format(modelname))

# Scaling the data. /255 since data is image data
X = normalize(X, axis=1)
print("({})".format(X.shape[0]/2400)+str(X.shape))  # (2400,50,50,1) - n,y,x,c

model = Sequential()
model.add(Conv2D(64, (3, 3), input_shape=X.shape[1:]))  # 64 3,3
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2))) #
model.add(Dropout(0.40))
model.add(Conv2D(128, (3, 3)))  # 5,5
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

# EXTRA
model.add(Conv2D(64, (3, 3)))  # 5,5
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

# remember to  flatten the data since the data is 2d and dense accepts 1d data
model.add(Flatten())
#model.add(Dense(128)) # 64 to 32

model.add(Dense(26))  # OG 1
model.add(Activation('sigmoid'))  # sigmoid

model.compile(loss='sparse_categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

# batch size should be kept a little low(20-200) to avoid negative results

model.fit(X, Y, batch_size=30, epochs=10, validation_split=0.2,callbacks = [board])  # OG 30
model.save("modelname")

i = 6
while i <= 10:
    model.fit(X, Y, batch_size=30, epochs=i, validation_split=0.2,callbacks = [board])  # OG 30
    model.save("{} - ({}).model".format(modelname,i))
    i = i+1

The error which I am getting is as follow:

python3 model_a-z.py
Traceback (most recent call last):
  File "model_a-z.py", line 16, in <module>
    board = TensorBoard(Log_dir="logs/{}".format(modelname))
    TypeError: __init__() got an unexpected keyword argument 'Log_dir'


texts: Hashing trick with random forest in scala

texts:
I am trying to perform a hashing trick and then a random forest with scala. I have the following code:

val documents: RDD[Seq[String]] = sc.textFile("hdfs:///tmp/new_cromosoma12v2.csv").map(_.split(",").toSeq)

val hashingTF = new HashingTF()
val tf: RDD[Vector] = hashingTF.transform(documents)

val splits = tf.randomSplit(Array(0.7, 0.3))
val (trainingData, testData) = (splits(0), splits(1))

val numClasses = 3
val categoricalFeaturesInfo = Map[Int, Int]()
val numTrees = 10
val featureSubsetStrategy = "auto"
val impurity = "gini"
val maxDepth = 8
val maxBins = 32

**val trainingData2=LabeledPoint(1.0,trainingData.collect())**

val model = RandomForest.trainClassifier(trainingData2, numClasses, categoricalFeaturesInfo, numTrees, featureSubsetStrategy, impurity, maxDepth, maxBins)

But I have the error

found   : Array[org.apache.spark.mllib.linalg.Vector]
 required: org.apache.spark.mllib.linalg.Vector

in the bold line.

Do you know how I can solve it?

Thank you,

Laia


texts: Why convolution over volume sums up across channels?

texts:
A simple question about convolution over volume .

Say we have an image with dimensions $(n, n, 3)$ and we apply a filter of dimension $(k, k, 3)$ this outputs an matrix of dimension $(n-k+1, n-k+1)$.

Why do we sum across channels in this case. Don't we lose information by mixing different channels. In case of images, this implies mixing information in R, G, B channels? For ex. when trying to detect traffic signal lights, such mixing can be fatal.


texts: NLP - extract sentence parts related to people

texts: Thank you for your help, I appreciate your time.


texts: How do I approach learning Data Science/ML the 'rightest' way?

texts:
First of all, I am not sure if this is the right place to post this so please do let me know if it isn't and tell me where it should be. I really just don't know where to go with this question.

Some backstory: I am going through my 2nd semester of my 2nd year of Software Engineering. I love math, I love computers, I love data and I love image processing so the most logical place for me to be is Data Science because I seem to be gravitating towards ML/CV a lot.

Thing is, I want to start learning Data Science/ML but I don't know where to start. Everyone keeps recommending online courses such as Andrew's coursera course but I have become somewhat skeptical of most online courses because they seem to love simplifying info too much that I feel like it just passes the threshold of it being useful info.

My assumption is: learning Data Science/ML/CV requires a very strong and rigorous foundation. I shouldn't start learning it with high level tools and little understanding of what makes it what it is. This is my problem. I don't know where to begin learning that way.

I searched for books and the two books I found people talking about were:

Pattern Recognition and Machine Learning, by Christopher M. Bishop. This book seemed to be very rigorous with a strong focus on the mathematics and intuition of things (looking at its index). Not much skepticism towards this one.
Hands-On Machine Learning with Scikit & TensorFlow, by Aurelien Geron. This one seems to be extremely popular and almost unanimously praised/liked/recommended by people but just looking at its name I felt a bit skeptical that it might have less focus on building strong foundation/intuition and more focus on using high-level tools.

I am open for other suggestions if you think some 3rd book/resource is more adequate for my need. I'd also greatly appreciate it if you briefly explain why you think it is more adequate.

Also it's worth noting that I have no problem with high-level tools and I know that I will eventually have to use them. I just feel like I'd be significantly better at what I want to do and valuable if I start from there.


texts: Interpolate function isn't returning values

texts:
My input DataFrame is structured like this:

I have a list of unique IDs (UIDs) which I would like to compare with each other, year on year (based on the "Age" column).
The challenge is, the data points don't align exactly, so I have generated new age values for each UID, for each integer up to 25 and appended these to the DataFrame.
Following this, I would like to interpolate the condition values for each of these ages, grouped by UID. This will allow me to tabulate (and compare) a condition value for each UID at each of these integer ages.
Below is what i've managed to come up with so far, I feel like i'm close to the solution but the condition values are not being populated. I don't get any errors when running the script but am obviously going wrong somewhere. I'd appreciate if someone could point me in the right direction!
import pandas as pd
#Read Interpolation input
df2 = pd.read_excel(r"Input_File_Path")

#Generate list of unique UIDs
UIDs =pd.Series(df2['UID'].unique().tolist())

#For each id, add ages 1-25 to the dataframe
interp_ages = []
for x in UIDs:
    for y in range(1,26):
        interp_ages.append(
                {
                    'UID': x,
                    'AGE': y,
                    'CONDITION': ''
                }
            )
interp_ages = pd.DataFrame(interp_ages)
df2 = pd.concat([df2, interp_ages])

#Sort values and set the index
df2.sort_values(by='UID', axis=0, inplace=True)
df2.index = df2["AGE"]

#Apply interpolation
df2.groupby('UID').apply(lambda group: group.interpolate(method='index', inplace = True))

#Export
df2.to_csv(r"Output_File_Path", index = False)
print("file exported")


texts:
I'm working with dynamic graph embedding but I can not seek the dataset for my problem. Can you help me find dataset with some requires:

A series of graph which represent for dynamic graphs. Each graph in series of graph is static graph with both directed and undirected
Label for each node (vertex) of graph to evaluate the embedding algorithm.

Thank for helping!


texts:
I have the following image:

I need to get two ellipses which will 'describe' my 'ring'.

I have the following code at the moment:

import cv2
import numpy as np
import imutils

image = cv2.imread('front2.png', cv2.IMREAD_COLOR)

# Convert to grayscale
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

_, gray = cv2.threshold(gray, 50, 255, cv2.THRESH_BINARY)

# Downsize image (by factor 4) to speed up morphological operations
gray = cv2.resize(gray, dsize=(0, 0), fx=0.25, fy=0.25)
cv2.imshow("Gray resized", gray)
cv2.imwrite('Gray_resized.png', gray)

# Morphological opening: Get rid of the stuff at the top of the ellipse
gray = cv2.morphologyEx(gray, cv2.MORPH_OPEN, cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5, 5)))
cv2.imshow("Gray removed noise", gray)
cv2.imwrite('Gray_removed_noise.png', gray)

# Resize image to original size
gray = cv2.resize(gray, dsize=(image.shape[1], image.shape[0]))

# Find contours
cnts, _ = cv2.findContours(gray, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)

# Draw found contours in input image
image = cv2.drawContours(image, cnts, -1, (0, 0, 255), 2)

if len(cnts) != 0:
    for cont in cnts:
        if cont.size < 10 or cv2.contourArea(cont) < 100:
            continue
        elps = cv2.fitEllipse(cont)
        cv2.ellipse(image,elps,(0,255,0),2)

# Downsize image
out_image = cv2.resize(image, dsize=(0, 0), fx=0.25, fy=0.25)
cv2.imshow("Output image", out_image)
cv2.imwrite('Output_image.png', out_image)

I've got images like:

In this way it detected one ellipse well enough but it hasn't detected the second one.

As i read it'd be more logical to use 'cv2.THRESH_BINARY_INV' in threshold but in this way it doesn't detect ellipse in the center:

Also if i try to lower the threshold my ring starts to fade:

The main questions: How to get rid of this noise at the top of my ring for detecting contours? And how to detect these ellipses?

I've looked at these answers:
1st
2nd
3rd
 but i haven't managed to achive my goal.

P.S. I can't use Hough-Circle method here. Because i have few similar images and radius for 'x' and 'y' axis may different and i need this precision(it means i need only ellipses and their radiuses for two axises).


texts: Lemmatization of pandas column using Wordnet after POS

texts:
I have a pandas column df_travail[line_text] with text.

I want to lemmatize each word of this column.

First I Lowercase the text :

df_travail ['lowercase'] = df_travail['line_text'].str.lower()

Then, I tokenize it and apply POS (because of wordnet default configuration which consider every word as noun).

from nltk import word_tokenize, pos_tag
tok_and_tag = lambda x: pos_tag(word_tokenize(x))
df_travail ['tok_and_tag'] = df_travail['lowercase'].apply(tok_and_tag)

Then I have the following : (extract of the entire df_travail['tok_and_tag'])

"[('so', 'RB'), ('you', 'PRP'), (""'ve"", 'VBP'), ('come', 'VBN'), ('to', 'TO'), ('the', 'DT'), ('master', 'NN'), ('for', 'IN'), ('guidance', 'NN'), ('?', '.'), ('is', 'VBZ'), ('this', 'DT'), ('what', 'WP'), ('you', 'PRP'), (""'re"", 'VBP'), ('saying', 'VBG'), (',', ','), ('grasshopper', 'NN'), ('?', '.')]"
[('actually', 'RB'), (',', ','), ('you', 'PRP'), ('called', 'VBD'), ('me', 'PRP'), ('in', 'IN'), ('here', 'RB'), (',', ','), ('but', 'CC'), ('yeah', 'UH'), ('.', '.')]

However, then, I'm lost about the lemmatization function to apply (with Wordnet), in order to take into account the fact that I applied POS ?


texts: RStudio/RMD files to create a student version assignment from the instruction version assignment solution

texts:
Does anyone know plugin or package in Rstudio to create a student version assignment from the instruction version assignment solution?

I know nbgrader for Jupyter Notebook that can do this, but I am working on RMD files in RStudio.


texts: Improving the pix2pix Architecture for Sketch to Image Translation on a Dataset of Sketches of People to Photos of People

texts:
For a university project I need to create a neural network which translates sketches of people into images.
In order to implement such a neural network, I decided to implement a pix2pix GAN architecture.
The neural network is trained an evaluated on a modified version of the CUFS dataset provided by my professors.
While the neural network is able to perform image transalations, as shown in the image pair below, I wonder if there are ways to improve the quality of the results?
In particular I was wondering if there are ways make the image look sharper?

I am already augmenting the relatively small training dataset of 70 sketch-photo pairs to 770 ones.
Furthermore, I tried to decrease the value of the $\lambda$ parameter.
Regarding the epoch, I figured out that the validation loss does not drop significantly after 160 iterations anymore.
I implemented the network in a Jupyter notebook hosted on Google Colab.
As you can see in the notebook's last cell, the currently achieved L1 loss on the testing dataset is about 0.191 on average.
Here is a link to the dataset used for training and testing.
Please feel free to take a look at the code and tell me ways on how to improve the code.
In addition, I would be most grateful if you could provide additional ways on how to improve the generated image quality.


texts: Difference between class_weight and loss_weights in Keras

texts: Keras has parameters class_weight used in fit() function and loss_weights used in compile() function. From the explanation(Docs) and what I understand, it seems that both are identical, as they are used to indicate the importance of each class. However, what is the difference between the two? And what are the scenarios in which they can be appropriately used?


texts: Shallow Neural Net for predicting numbers other then 1 or 0?

texts:
I'm not gonna lie, I'm very new to neural networks but am also so interested in them and learning the way they work and what can be made from them. So in my endeavors for learning, I stumbled upon Siraj Ravel's youtube channel and furthermore, his Github where he posted this 1 neuron neural network that predicts the output given inputs.* I looked at it and his accompanying video and got the basic gist of how it works. After running it a few times making minor adjustments to say the inputs, I decided why not try to add numbers other than 1 and 0. I made a pattern with the output being the inputs added, however, I ran into an error where the output of its test was '1.'. I assumed this was because of the sigmoid function forcing it to be in between 1 and 0.

Summary: How can I change that program so I can give it an input of patterns with different numbers than 1 and 0 and it will output correctly (examples below)

*In his program inputs were like [1, 0, 1] = 1, [0, 0, 1] = 0 and [1, 0, 0] = 1 (pattern is first column of matrix is answer) and he asked the program to give output for [1, 1, 0] and it correctly outputted 1.

*My goal for it was so given "[2, 2, 2] = 6, [1, 2, 0] = 3, and [1, 1, 2] = 4 (more training included if necassary)" as input could it get 4 as output for "[2, 2, 0]?

Code for program


texts: How to fix internal folds for parameter tuning in random forest and SVM?

texts: I have implemented logistic regression using glmnet library. For hyper parameter tuning internal cross-validation is performed. I can fix the internal folds using set.seed so that foldid parameters can be given. I want to compare the predictive performance in SVM and Random Forest classifier. I want to know If there is a similar parameter in Random Forest and SVM so that I can fix internal folds for hyper parameter tuning in these functions.


texts: Appropriate ML algorithm for allocating students to seats

texts:
I'm new to machine learning and am trying to use it to solve a specific problem related to seating students in a classroom. I want to take a list of students and allocate them each to a seat, such that a certain output value is maximised.

An example:

Students (each one containing all the data from which a compatibility score can be calculated):
A
B
C
D
E
F

Seats:
1
2
3
4
5
6
7
8

Each pair of students has a predefined 'compatibility score' and each pair of seats has a geometric distance between them. After sorting students into seats, I would calculate a ratio between those two values, the average of which would be my outcome to be optimised.

A class of 25 students and 25 seats has something like 15,000,000,000,000,000,000,000,000 possible arrangements, so a brute force method is rather unfeasible. My ideal outcome here is that machine learning can develop an optimised algorithm for sorting them.

Any ideas for what kind of ML algorithm I want?


texts:
How do I perform frequency/count encoding for a train and test set?

The implementations of this encoding I've seen simply frequency encode the categorical variables on a particular dataset (no specific train, and test encoding transformation). For instance:

dataset.groupby("cat_column").size()/len(dataset)

In my case now I have a train, and test set.

[First option] Is it okay (due to leakage? or there won't?) for me to use frequency encoding on the whole dataset. OR

[Second option] I should take into consideration train, and test set independence.

If the second option, how do I do this?

Encode the train set, then use the encoding values of categories in the train set for the same categories in the test set. Categories not represented in the test set would be need to be handled. OR
There's a better generic implementation?


texts: What should be the training frequency of a rnn model for timeseries prediction?

texts: If I use a rnn model for time series forecasting how frequently do I have to retrain the model.


texts:
I'm wondering how does ndimage.measurements.center_of_mass calculate the center of mass values for a two dimensional numpy array.
For example let's say we have an array m:
m = numpy.array([[1,2,3,1],[4,4,4,1],[5,6,7,2]])

Then one would get the following for the coordinates of the center of mass (x, y):
x, y = ndimage.measurements.center_of_mass(m)

x = 1.325  and  y = 1.3

So, how are those values of x and y mathematically derived (actual derivation)?
When I follow the steps in the function definition center_of_mass, I don't get that result.


texts: Subtract Rows of Matrix from rows of another matrix numpy

texts:
I have two matrix V_r of shape(19, 300) and vecs of shape(100000, 300). I would like to subtract rows of V_r from from rows of vecs. Currently, I am achieving this with the following code. Is there a way to do it using broadcasting?

    list=[]
for v in V_r:
    a=np.linalg.norm(vecs-v,axis=1)
    list.append(a)
M=np.vstack(list)


texts: I am new to this data mining. Can anyone please help me with an example for Sliding Window Filtering algorithm(SWF) for incremental mining?


texts: Word2Vec running on Local machine(Mac OS) but not on AWS Instance

texts:
I used word2vec on a customer review dataset provided by my company and ran it on my local machine - Mac OS Sierra - 10.12.1(Beta) with 8gb RAM. It is working fine. But I wanted to fasten up the process and henceforth tried to run it on an AWS Instance with 16gb RAM. But when I tried to run it there, the program keeps crashing with memory issues or segmentation faults. I tried to provide 16gb of swap memory more. Even that memory gets full and the program crashes.

Am I missing to do something here?
Why can't the memory be handled by the server like my MAC?
Can I do something so that the program successfully runs on my server too?

Note: The code and the dataset is the same on both local machine and the server.


texts: Tagged dataset with photos for race detection

texts: Looking for the tagged dataset, because I would like to identify race by photo. I tried using the UTKFace dataset from Kaggle, but it outputs hispanic and Arab people on images as other. My customer would like those ones as separate categories. Do you have any datasets like that? Because UTKFace has only 4 tags - black, white, indian, other.


texts:
I want to ask wich algorithm can I use to do a sequences classification , knowing that I have two classes (positive /negative), but training is done using data from one class only (positive).

Thank you


texts: ValueError: Error when checking target: expected dense_1 to have shape (7,) but got array with shape (1,)

texts: Hi everybody can somebody help me!! I don't know what's wrong in my code and why it gives me this error ???


texts: how to reshape xtrain array and what about input shape?

texts:
from keras.datasets import mnist
from keras.layers import Activation,Dense,Convolution2D
from keras.models import save_model,load_model,Sequential
from keras.callbacks import TensorBoard
import matplotlib.pyplot as pl

(xtrain,ytrain),(xtest,ytest)=mnist.load_data()
model = Sequential()
model.add(Convolution2D(32,3,activation='relu',input_shape=(60000,28,28)))
model.add(Dense(10, activation='relu'))

model.summary()

model.compile(optimizer='rmsprop',
              loss='categorical_crossentropy',
              <space>metrics=['accuracy'])

model.fit(xtrain, ytrain, epochs=10, batch_size=60000)


texts:
When talking about artificial neurons, inputs, weights and biases, I understand the role of each but the latter.

In short if we have a neuron such as sigmoid(sum(w*x) + b) I get that the weights basically say which of the inputs is more "important", but what about the bias? I've read it in this other question "is as means how far off our predictions are from real values."

But how can this be true, if we start them "at random"? Also this isn`t supposed be be the job of the loss/cost function?


texts: What is an object detection problem with only one class called?

texts:
Object detection is defined as the problem in which a model needs to figure out the bounding boxes and the class for each object. A lot of ML solutions for object detection base around having "two passes" - one for creating the bounding box of the region and another for classifying it.

I was wondering if there is a name for a subset of this problem where $n_{classes} = 1$.  I feel like there is an interesting opportunity here as the whole classification part of the model can (basically) be ignored. Obviously, I can just train a typical object detection model with one class, but was just interested to see if there are any more specialized methods.


texts: I'm looking for methods of Community Detection in networks. For example if I have a network of 100 people (each node is a person), how can I cluster nodes? What would be the best approach for grouping these People? I know this question is rather open, but I'm just looking for a nudge in the right direction.


texts: Building a prediction model for dynamic coordinates and later categorize as binary classification

texts:
Project Summary

I have a website (academic project) that records mouse movements such as click, mouse up, mouse down, etc. It records the coordinates for each event on a given web page from a visitor.

What am I trying to accomplish?

I would like to do a dynamic polling per 100 msec from a website and compare if those points/graphs/velocities etc of user movement already existed in ML Model or not. Each time it polls, it will check against the ML model and will give me the confidence of the matching values. So, this will be an online ML model which will poll against multiple models that I have.

What I do not know and where to start?

In the first approach, I was brainstorming: Would it be better to have each 100 msec during polling of the coordinates to draw a graph and compare it against the models that I have? The idea that I got was image similarity matching. Each 100 msec will draw a dynamic graph and compare on models that I have allocated against each polling.
The second approach that I was brainstorming: Have velocities of coordinate changes into the time-series graph and compare the slice of the graph for the given time series (T-current). Here, the graph will be sliced and compared against the time series.

Expertise seeking summary

I would like to identify, with the dynamic mouse movement pattern, that this pattern was already in the ML model or not in an online mode. This will be handled prefetching the information from the already trained model. It would be great if you could point me in the following suggestions:

What kind of models or ways to handle this example would be best suited? Example of links that support this.
Real-time modeling interpretation/classification using the already built modeling

References - These are the tutorials that looked so far

https://www.vis.wiwi.uni-due.de/uploads/tx_itochairt3/publications/cgf-dynamicgraphs.pdf
https://ieeexplore.ieee.org/abstract/document/4734024
https://www.sciencedirect.com/science/article/pii/S2468502X20300140
https://ieeexplore.ieee.org/document/5190876
https://scholarworks.umass.edu/cgi/viewcontent.cgi?article=1906&context=dissertations_2
https://stackoverflow.com/questions/5730631/image-similarity-comparison
https://towardsdatascience.com/prediction-and-analysis-of-time-series-data-using-tensorflow-2136ef633018
https://towardsdatascience.com/how-not-to-use-machine-learning-for-time-series-forecasting-avoiding-the-pitfalls-19f9d7adf424
https://medium.com/@akarshzingade/image-similarity-using-deep-ranking-c1bd83855978
https://medium.com/dataseries/easily-deploy-deep-learning-models-in-production-13db48071578


texts: How do I use Convolutional Neural Nets to classify if there are rain drops or soil on the camera lens?

texts:
I have certain videos for which the frames are labeled either as dirty (meaning the camera lens is occluded by soil or rain) or as clean.
The goal is to test a convolutional neural net on this data to evaluate how well it can classify if a frame is dirty or clean.

One idea is to use the first layers of an existing network and train last (fully connected) layers using the available data.

But one issue might be that basically all available networks are trained for object classification and might not be very suitable for tasks like soil and rain detection.

Do you have any recommendations about networks or models which might be suitable for this task?


texts: Are there machine learning methods that can give captions to the New Yorker magazine cartoons?


texts: Is sequential pattern mining possible with machine learning?

texts: I am curious if sequential pattern mining algoritmhs fill a unique gap or the same thing can be achieved with alternative methods for example wih machine learning or something else. Do you know any alternative methodology that can achieve the same thing? (For me this would be relevant, because if there is an alternative method I can compare them in my thesis.)


texts: How to use features when predicting aggregation?

texts:
Let's say I have to predict the total monthly sales of a store. I have the data in the following format:

I don't think I can use this data directly for predicting the sales for a month, as in it's current form there is no target variable present to predict. We would have to sum the sales for each month and only then will we get our target variable (the sales that we want to predict).

Please see when I group the sales by month (Jan and Feb):

Now the above is data I can use with ml algos to predict the Total_Sale which is my target variable. But, since now I have grouped the sale by month, I have lost all the features like Gender, Age, Product etc... I cannot explain these features and see their impact on the sale.

Is there any way around this?

Thanks


texts: Transfer learning with many small datasets

texts:
Context
I am working on a NLP-model that can classify documents into one of N categories. I have document data from a number of different customers. The document topics are similar across customers but they classify them into different categories.
For simplicity, assume that the documents can contain six different topics: A,B,...,F. Each customer classify the documents differently from the topics, i.e. N mentioned above is customer specific and the mix of topics is (somewhat) different: Customer 1 have three classes (ABC, D, EF), customer 2 have four (A, BC, DE, F), customer 3 have four (AB, CD, E, F), and so on.
The goal is to use the data from the current customers to develop a model that can transfered and retrained on a new customer (with a different mix of topics A,B...,F and potentially only a small amount of data)
What I have done
I have been searching here and in papers for strategies on transfer learning with multiple smaller datasets with different output categories but have not been able to find anything. I have only been able to find papers that deals with larger datasets.
What I am considering
My current thought is to use a deep learning (keras/tf) model with this layout:
document(from customer X) -> preprocessing(shared among customers) -> feature extraction (shared among customers) -> classification (specific to customer X)
With this approach, I would during training switch the customer specific classification layer for each customer. This would train the preprocessing and feature extraction layers on data from all customers but only train the classification layers on individual customer data.
Questions

Do you have any references to papers or any other material that describes a similar situation?

Is this the best strategy for training this model? More specific, is it possible to make a layout of the model where the all customers classification layers are present in the model at the same time and the switching between layers can be handled by a cleverly engineered loss function?


texts: How to choose the best optimiser in Deep Learning algorithms?

texts:
I have been training a CNN to classify 4 faulty (acoustic emission/250KhZ) signals. I have no problem in implementing the algorithm using tensorflow libraries, but I am confused with which optimizer to choose. I have been trying out different optimizers, for Adam optimizer and the gradient descent optimizer algorithm gave the same results in test classification accuracy(~84% to 85%) on k-fold cross-validation. Results are same for both optimizers, is it because of the data or because of training on huge data(64000 samples/fault condition)?

Are there any specific guidelines to choose a particular optimizer for kind of data? Or else, is it randomly by trial and error?


texts:
I'm currently facing a project that I could solve with a relational database in a relatively painful way. Having heard so much about NOSQL, I'm wondering if there is not a more appropriate way of tackling it:

Suppose we are tracking a group of animals in a forest (n ~ 500) and would like to keep a record of a set of observations (this is a fictional scenario).

We would like to store the following information in a database:

a unique identifier for each animal
a description of the animal with structured fields: Species, Genus, Family, ...
a free text field with additional information
each time-point at which it was detected close to a reference point
a picture of the animal
an indication whether two given animals are siblings

And:

there might be additional features appearing later as more data comes in

We would like to be able to execute the following types of queries:

return all the animals spotted between in a given time interval
return all the animals of a given Species or Family
perform a text search on the free text field

Which particular database system would you recommend ? Is there any tutorial / examples that I could use as a starting point ?


texts: Computing Attention on multi-dimensional sequences?

texts:
Is it possible to compute attention/adapt existing transformer architectures (like longformer) to be used on multi-dimensional sequence input?
As in, instead of a 1D array of tokens (like a python list of tokens to be used to calculate attention on), I feed an array of 2D/3D/4D tokens and I want to pre-train my language model on that via Masked Language Modelling technique (i.e predicting masked tokens).
Is it even possible to do this? any idea what modification I would have to make?


texts: OneHotEncoder in ColumnTransformer: passthrough not returning original column names

texts:
I'm using a column transformer to pass in categorical data of the Kaggle Titanic dataset to one-hot encode. I'm using a pipeline as well so I can expand the process later on, but the dataframe column names returned are completely different than what was input:
"""
training set info:
 #   Column    Non-Null Count  Dtype
---  ------    --------------  -----
 0   Survived  889 non-null    int64
 1   Pclass    889 non-null    int64
 2   Sex       889 non-null    object
 3   Age       889 non-null    float64
....
"""

dt_ct=ColumnTransformer([('onehot',OneHotEncoder(),['Sex','Embarked'])],remainder='passthrough')
dt_pipe=Pipeline([('col_trans',dt_ct)])
arr_out=dt_pipe.fit_transform(train_set[[x for x in train_set.columns if x!='Survived']])

dt_pipe['col_trans'].transformers_
"""
returns:
[('onehot', OneHotEncoder(), ['Sex', 'Embarked']),
 ('remainder', 'passthrough', [0, 2, 3, 4, 5])]
"""

I am able to get the new categorical names of Sex and Embarked using 'get_feature_names()' but how would one get the rest of the column names that were passed through into the column transformer ('remainder')?


texts:
I'm building a neural network for data analysis, however I'm stuck on how many output neurons I need and what they should represent. The neural network tries to predict peoples choices for certain objects. There are 75 possible objects, but here's the catch, they choose 6 of them at a time. These six can (theoretically, though this won't happen realistically) be any combination of the 75 objects, in rare cases there might even be duplicates of a certain object in this set of six.

I naturally considered creating an output neuron for each possible set, but since that would lead to 75^6th output neurons (or more if we consider duplicates) I imagine that that would have disadvantages for the learning speed. Another option I considered is just taking the six highest ratest items as a set, but I'm unsure if this would work since the choice of the second, third, etc. objects depend on which earlier objects were chosen. I wonder if there are any better, faster or more accurate ways of doing this?

Edit, some more information about the trainingdata:
The neural network will be used to analyse the ingame item choices in the computergame called league of legends. There players can choose up to six items each game. These six items are most likely based on some factors that change each game such as the ingame characters they and their enemies play, but also things such as what the other 5 items are they have. However, the reason for this may be universal to all games. For instance, the (likely) biggest reason why items depend on eachother is the multiplicativity of item stats, if you already have an item that makes your attacks hit harder then getting an item that allows you to attack more often is going to do more for you than getting an item that makes your attacks hit even harder. There are also some links between item choices though, while probably more limited, there is for instance an item that returns damage to the enemy when he attacks you, while it's not detrimental it is probably not a great idea to combine that with an item that lowers the amount of attacks that an enemy can do to you.


texts: Should I divide my dataset in multiple prediction models?

texts:
Let's say I have an expertise in the domain knowledge of the dataset I am working on. I know that part of my dataset acts 100% differently than the rest.

Also, it is straightforward to check if a row belongs to one part of the dataset or the other with a couple of simple if-else.

Should I split my dataset upfront and create two different prediction models for the two parts of the dataset? Or should I keep one model and try to improve it?

Then when I want to use the model for predictions, I check the input values and see in which category it belongs and I call the appropriate model for it.


texts:
I have a 500MB model which I am commiting to Git. That is a really bad practice since for newer model versions the repository will be huge. As well, It will slow down all builds for deployments.

I thought of using another repository that contains all the models and then fetch them in running time.

Does anybody know a clean approach or alternative?


texts: Why is r squared lowered when adding polynomial features?

texts:
I am trying to find a best fit line f(x) = ? for a random set of x,y coordinates.

Linear Regression with polynomial features works well for around 10 different polynomials but beyond 10 the r squared actually starts to drop!

If the new features are not useful to the Linear Regression I would assume that they would be given a coefficient of 0 and therefore adding features should not hurt the overall r squared.

I reproduced this problem when housing price predictions when creating a large amount of interaction features.

I have my python code below:

Create Random Data

import numpy as np
import matplotlib.pyplot as plt

def pol(x):
    return x * np.cos(x)

x = np.linspace(0, 12, 100)
rng = np.random.RandomState(1234)
rng.shuffle(x)
x = np.sort(x[:25])
y = pol(x) + np.random.randn(25)*2

plt.scatter(x, y, color='green', s=50, marker='.')

plt.show()

Regress and Check Each R Squared

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures

for p in range(1,30):
    plot_range = [i/10 for i in range(0,120)]
    poly = PolynomialFeatures(p)
    X_fin = poly.fit_transform([[samp] for samp in x])
    X_fin_plot = poly.fit_transform([[samp] for samp in plot_range])
    reg = LinearRegression().fit(X_fin, y)

    from sklearn.metrics import mean_squared_error, r2_score
    print(p,r2_score(y, reg.predict(X_fin)))

Display Last Regression Line

plt.scatter(x, y, color='green', s=50, marker='.')
plt.plot(plot_range,reg.predict(X_fin_plot))
plt.show()

I also have two plots to compare. The first is with 10 polynomial features and the second is with 40. Notice how the second misses the majority of the first points.


texts:
I was training a CNN when the following appeared:

after this the training continue, but I don't understand why it happens. Should I do something or leave it?

[EDIT] The problem is that it does this also during the evaluation, so for example if I want to print the confusion matrix:

Y_pred = model.predict_generator(test_generator, test_generator.n //
batch_size+1)
y_pred = np.argmax(Y_pred, axis=1)
print('Confusion Matrix')
print(confusion_matrix(test_generator.classes, y_pred))

I get the error:

Thanks.

[EDIT] I have 3200 images for training and 800 images for testing. So I don't understand what is happenining here because this error should happen for the fact that y_test and y_pred have different number of elements, so why it says that one has 832 elements, since I have only 800  images for testing?


texts: What is the difference between the 3 architectural components of Microsoft R Server?

texts:
Lately, I've been experimenting with Microsoft R Server(MRS) and SQL Server R Services. I've come across that MRS is built on architectural components like ScaleR algorithms optimized for fast parallel execution, DistributedR parallel computing framework for managing compute resources, and ConnectR for versatile connections to data sources.

I've also learnt (and got confused) that

ScaleR algorithms are basically functions of the RevoScaleR package built on the DistributedR framework.
ConnectR component of the architecture is also a bunch of RevoScaleR functions / adapters to access a variety of data sources.

My question is:
What is RevoScaleR in relation to DistributedR and ConnectR?

Where does DeployR fit into all this?

In general, I want to know the high level differences between
RevoScaleR package
DistributedR
ScaleR
ConnectR
DeployR


texts: Trouble in calculating the covariance matrix

texts:
I'm trying to calculate the covariance matrix for a dummy dataset using the following formula, but it's not matching with the actual result.

Let's say the dummy dataset contains three features, #rooms, sqft and #crimes. Each column is a feature vector, and we have 5 data points. I'm creating this dataset using the following code:

matrix = np.array([[2., 3., 5., 1., 4.],
                   [500., 700., 1800., 300., 1200.],
                   [2., 1., 2., 3., 2.]])

Let's normalize the data, so the mean becomes zero.

D = matrix.shape[0]
for row in range(D):
    mean, stddev = np.mean(matrix[row,:]), np.std(matrix[row,:])
    matrix[row,:] = (matrix[row,:] - mean)/stddev

Now, I can write a naive covariance calculator that looks at all possible pairs of features, and that works perfectly.

def cov_naive(X):
    """Compute the covariance for a dataset of size (D,N)
    where D is the dimension and N is the number of data points"""
    D, N = X.shape
    covariance = np.zeros((D, D))

    for i in range(D):
        for j in range(i, D):
            x = X[i, :]
            y = X[j, :]
            sum_xy = np.dot(x, y) / N
            if i == j:
                covariance[i, j] = sum_xy
            else:
                covariance[i, j] = covariance[j, i] = sum_xy
    return covariance

But, if I try to implement the formula mentioned in the beginning, the result is incorrect. The method I am trying out is as follows:

def cov_naive_2(X):
    """Compute the covariance for a dataset of size (D,N)
    where D is the dimension and N is the number of data points"""
    D, N = X.shape
    covariance = np.zeros((D, D))

    for i in range(N):
        x = X[:, i]
        covariance += x @ x.T
    return covariance / N

What am I doing wrong here?

Expected output:

array([[ 1.        ,  0.96833426, -0.4472136 ],
       [ 0.96833426,  1.        , -0.23408229],
       [-0.4472136 , -0.23408229,  1.        ]])

Actual output from cov_naive_2

array([[3., 3., 3.],
       [3., 3., 3.],
       [3., 3., 3.]])


texts:
I have a data frame which has three columns as shown below. There are about 10,000 entries in the data frame and there are duplicates as well.
Hospital_ID   District_ID  Employee
Hospital 1    District 19   5
Hospital 1    District 19   10
Hospital 1    District 19   6
Hospital 2    District 10   50
Hospital 2    District 10   51

Now I want to remove the duplicates but I want to replace the values in my original data frame by their mean so that it should look like this:
Hospital 1    District 19   7.0000
Hospital 2    District 10   50.5000


texts:
What is intuition behind multimodal distribution? and
How does GANs generate samples from it?


texts: Use correlation matrix scores as starting weights and bias inputs for neural network?

texts:
I have a neural net that's generating an average 15% error across the three outputs it gives. My problem is two of the three really have about a 2% error while the third has around 40%.

I was wondering if anyone has used results from a correlation matrix on the raw data to set starting parameters for what features are valued more relative to the labels. If this is possible, or logical, how would you initialize this for two input features if your model's first layer has say eight nodes - or at least more than two? I'm using Keras btw, but even a theoretical explanation would be helpful.

If it doesn't make sense to follow this path, does anyone have recommendations at correctly the imbalance?


texts:
I'm trying to tackle a classification problem with a neural net tensor using flow.
I have some continuous variable features and some categorical features.
The continuous features are normalized using sklearn's StandardScaler.
For the categorical features I am using a series of embedding features that I'm concatenating together with my continuous features.

The embedding features are created like so :

 airline = tf.feature_column.categorical_column_with_hash_bucket(
        'AIRLINE', hash_bucket_size=10)

then :

 tf.feature_column.embedding_column(airline, 8)

However I am having trouble choosing my embedding feature output size. I understand this transforms my sparse one hot encode "airline" feature into a float vector of size 8.

Is there a heuristic I can use to choose an embedding feature size ?

My neural net's accuracy remains stuck at 31%. It doesn't seem to be learning even after a 100 epochs. Could the size of the embedding features be a cause for such a behaviour ?


texts: Interpreting coefficients after transforming Y (dependent) variable

texts:
I have been working on linear regression for forecasting purposes.

I have a model where I have used a BoxCox transformation on the $Y$ variable (Sales) with $\lambda = 0.3$. The model is as follows:

$$Y = 13 - 0.2 X_1 + 3 X_2$$

How can I interpret the effect of sales from each variable, given the transformation?

For instance, assume $X_1=5$ and $X_2=0.3$ then we get:

$$Y = 13 - 0.2 \times 5 + 3 \times 0.3 = 12.9$$

Reverse BoxCox of 12.9 with $\lambda = 0.3$ is 196. So it forecasts 196 sales. But if we increase $X_2$ from 0.3 to 1 and increase $X_1$ from 5 to 10 then we get:

$$Y = 13 - 0.2 \times 10 + 3 \times 1 = 14$$

Reverse BoxCox then gives 244 sales forecasted sales. An increase of 48 sales. But how much of that is down to $X_1$ and $X_2$?

If we calculate them separately the answer doesn't add up!

EDIT: to make the above point clear:

If $X_1$ went from 5 to 10 but $X_2$ doesn't change:

$Y = 13 - 0.2 \times 10 + 3 \times 0.3 = 11.9$, transform to 159 sales. A -37 decrease from the starting position (196-159)

However, if $X_2$ went from 0.3 to 1 but $X_1$ doesn't change:
$Y = 13 - 0.2 \times 5 + 3 \times 1 = 15$, transforms to 294 sales. A +98 increase in sales (294-196).

But if $X_2$ increases sales by 98 and $X_1$ decreases sales by 37 you would expect the net increase to be +61 when you include both changes. But from earlier we found that by applying both we actually get a +48 increase in sales!

How can we work out the actual sales change resulting in changes in $X_1$ and $X_2$?


texts: Prediction of output as 1 D vectors from multivariate time vector inputs

texts:
I have a data set where data in form of time sequences(which is evenly sampled at 1 sec).
For each set I have 4 inputs and 1 output. The length of input and output vectors is same. The entire data has around 5k such data samples.
I want to build a model where I can provide the time vectors and get the output as time vector of same length.
How can I build model for such cases. Can anyone guide me on how to start.
Thanks


texts: What is industry standard for workflow when training models?

texts: I'm assuming that each time someone trains a model and wants to tweak it/iterate, that they don't have to wait hours and hours for it to learn and then output. So my question is, how do people manage this workflow?


texts:
I just trained my first model in Python 3.7/scikitlearn (Linear Regression) (well I copied most of the code but its something ^^).

Now I want to actually Use the model. Specifically its about sons heights incorrelating to their fathers. So I now want to enter a new Father-height and get a predictions for its sons height.

How could something like this look like?

I read about "Pickle" to save a model and use it later, seems awsome but how would I use such a saved model?

If anybody can give me a simple answer or even just a link to atutorial would be great. Below is a piece of "my" code just for some context.

#Spliting the data into test and train data
X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2,random_state=0)

#Doing a linear regression
lm=LinearRegression()
lm.fit(X_train,y_train)

#Predicting the height of Sons
y_test=lm.predict(X_test)
print(y_test)


texts: How to generate a sample from a generative model like a Restricted Boltzmann Machine?

texts:
I am learning about the Boltzmann machine. So far, I have successfully written a code that can learn the coefficients of the energy function of a Restricted Boltzmann Machine. Now, since my model is generative (if I have understood things correctly so far) and I know for sure that RBMs can be used for inpainting in binary images at least, I want to know how I can generate a sample from my probabilistic distribution given by the Boltzmann machine. That is how a new binary image based on the training dataset is generated by my model.

However, even though mathematically the idea seems clear, I do not know how I can program a computer to sample from a distribution.


texts: XLNET how to deal with text with more than 512 tokens?

texts:
From what I searched online, XLNET model is pre-trained with 512 tokens, and https://github.com/zihangdai/xlnet/issues/80 , I didn't find too much useful information on that either.
How does XLnet outperform BERT on long text when the max_sequence_length hyperparameter is less than 1024 tokens ?


texts: pix2pix GAN with Rectangular Image Dataset

texts:
I am currently working on a project (for university) which translates sketches of faces to images of this person.
For implementing this, I decided to use a pix2pix GAN architecture.
However, I have the issue that the dataset contains photos in the format of 200x250 pixel instead of the 256x256 pixel the TensorFlow reference implementation is designed for.
The dataset is a modified version of the CUFS dataset provided by my professors.
I identified the following three possibilities for managing this dataset:

Rescaling all images to 256x256 pixel which I implemented as a first draft and which works quite okay,
pad the images to size 256x256 pixel and accept that parts of the training effort goes towards learning pixel which will always have a value of 0,
or to modify the architecture in order to process images of size 200x250 pixel, which was suggested by my professor.

I have implemented the first two alternatives, but I would like to also try to implement the third alternative.
However, I am kind of afraid I might break the architecture by randomly changing the dimensions of the convolution layers.
Thus, I want to ask you if you have any suggestions on how to adapt the architecture for inputs of size 200x250 pixel?
The source code is available as a Jupyter notebook hosted on Google Colab.
I also provide a link to the dataset, in case you need it.
Just for your information, I have to use TensorFlow and Keras for this project.


texts: Where to find list of Tensorflow pretrained models available in download.tensorflow.org/models

texts:
I am trying the find the pretrained models (graph.pd and labels.txt) files for Tensorflow (for all of the Inception versions and MobileNet)

After much searching I found some models in, https://storage.googleapis.com/download.tensorflow.org/models

like, https://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip

https://storage.googleapis.com/download.tensorflow.org/models/inception_dec_2015.zip

But I found this as links from various blogs/etc. But is there a way to list all the models/zips in this dir? https://storage.googleapis.com/download.tensorflow.org/models

I want the latest of Inceptionv3 and MobileNet, does anyone know the file names in models for these?


texts: modelling multirotor aerodynamics using datalogs of flights

texts:
I am trying to find a vector that would describe the effects of wind on a multirotor. I have a bunch of datalogs from a single frame of multirotor and am of the mind to digg.

The idea is that during flight a multirotor has 2 vectors to fight against:

Gravity - to stay in the air and level height
Wind

In a world without wind the multirotor would only fight against gravity when hovering and when flying in some direction. Each command the pilot makes is realized as gravity would dictate.

When we add wind, a certain deflection to multirotor angles occurs. If the wind is constant then the deflection is a function of desired movement and the wind parameters (strength and angle).

My idea is that across many flight logs the wind changes. You get average winds of different angles and different strengths. I would like to average out the wind effects and find a model that is the expected flight pattern when there is no wind. Using that model I would estimate the wind vector and using that knowledge I could do all sorts of clever stuff like choosing optimal flight speed.

So far I have worked on creating a database of data from the logs. Now I need to figure out how to proceed. How do I proceed?

I was thinking of trying to separate the positional (angles, height, speed, current consumption) data from the database and the desired parameters (direction, height, speed). Can this work as a sort of a set of labled data? The angle and current consumption show the effort needed to achieve the desired values. The multirotor algorithms try to be as close to the desired as possible and thus taking the angles and consuming as much current as is optimal in that point in time.

I would already be happy if I could label the wind strength into a few different levels to rough calculations for speed and other things. Levels like none, low, medium, high, toooo much to fly really.

Any pointers are welcome and since I am not well versed with this subject any keywords that I could read up on are much apreciated.


texts: How to deal with a small dataset for image classification using CNN?

texts:
I have a dataset consisting of characters(lowercase and uppercase) and numbers, totalling about 62 classes. The data I have are about 45 images per class and no test data. The data is a subset of the EMNIST dataset but we aren't allowed to use transfer learning. I am also using data augmentation on the test data. I've used train_test_split with (0.1) test data. I have been experimenting a lot with various CNN architectures but the validation accuracy is coming around 70% with testing accuracy sometimes going up to 90% but I guess it's mostly because of over-fitting.
How do I ensure I am able to make a robust model, or that the model I have made is feasible? Are there some methods I can employ to ensure that the model is able to deal with the small data well?
Also, the images are of size (900,1200) which I have resized to (64,64). Is there a general CNN architecture for this image size that I can use as a base for experimenting?


texts: What is the difference between LeakyReLU and PReLU?

texts:
I thought both, PReLU and Leaky ReLU are
$$f(x) = \max(x, \alpha x) \qquad \text{ with } \alpha \in (0, 1)$$
Keras, however, has both functions in the docs.
Leaky ReLU
Source of LeakyReLU:
return K.relu(inputs, alpha=self.alpha)

Hence (see relu code)
$$f_1(x) = \max(0, x) - \alpha \max(0, -x)$$
PReLU
Source of PReLU:
def call(self, inputs, mask=None):
    pos = K.relu(inputs)
    if K.backend() == 'theano':
        neg = (K.pattern_broadcast(self.alpha, self.param_broadcast) *
               (inputs - K.abs(inputs)) * 0.5)
    else:
        neg = -self.alpha * K.relu(-inputs)
    return pos + neg

Hence
$$f_2(x) = \max(0, x) - \alpha \max(0, -x)$$
Question
Did I get something wrong? Aren't $f_1$ and $f_2$ equivalent to $f$ (assuming $\alpha \in (0, 1)$?)


texts: Transfer learning (on pre-trained inception net model) for multi label classification is giving similar probability for all labels

texts:
Number of labels: 1000, Dataset size: 200000 images
Final probability for 1000 labels is in the range of 0.3 to 0.34.
I was expecting large variation in probabilities. Can someone tell me what I am doing wrong. I am following this tutorial


texts: Why is a normalized dependent variable important for model?

texts: If we have a highly skewed dependent variable, is it good practice to remove the outliers to force the data into a more normalized shape? If so, why is this important?


texts: How to detect phrases from an English sentence.

texts:
The question is not about detecting keyphrases. It is about detecting a combination of words makes a valid phrase or not. For example,

"John reads New York Times in New York."

Here, the phrases are

New York Times
New York

Detecting keyword phrases is a Text Summarization problem, however, here it is classifying whether a combination of words make a valid phrase or not.
There have been a few algorithms we have gone through including but not limited to models.phrases – Phrase (collocation) detection of Gensim, however, we are looking for better results.


texts:
I'm completely lost when trying to choose the type of predictive model for my problem. Is it autoregressive model, nonlinear time series, Markov Chain or other? Can someone please give me some advise?

78,
18,
51,
89,
19,
43,
62,
28,
94,
49

Suppose, everyday I'm given 10 data, and an example was listed above. They're random numbers generated by two devices, namely Device A and Device B. Each of them is capable to generate random numbers from 0 to 9.

The first number in the data is generated by Device A, while the second number is generated by Device B. For instance, for the first data of "78", "7" was generated by Device A and "8" was generated by Device B. Similarly, for the last data of "49", "4" was generated by Device A, and "9" was generated by Device B.

I want to be able to predict the next outcome variable after the last "49".

I have a total of 300 historical data for 30 days.

From my initial investigation for the 300 data, every device tends to produce repeated sequences. For instance, Device A will repeat the sequence "6-2-9-4" (as in the last 4 data). That means this sequence appeared twice within the 300 historical data for Device A. For another example, the sequence "8-1-9-9" (the 2nd to the 5th data) in Device B appeared twice, too. Each of them produce at least three repeated sequences.

I'd like to predict the next outcome variable after the last "49". Which model is more appropriate?

Thank you in advance!


texts: Calculating the lower and upper bounds forVC-dimension of a decision tree

texts: I have a problem finding the lower and upper bounds of the decision tree. Suppose there is a decision tree with a hypothesis space of depth 2 and an input space with 10 variables (the variables take boolean values). How do I calculate the lower and upper bounds for the VC dimension of this tree? I was doing some reading and papers but couldn't actually establish a strong understanding of the technique. Even some answers on stack exchange were confusing. I would appreciate it if someone can help me understand this clearly.


texts:
I implemented Isolation Forest(Liu et al. 2008), but when testing with different datasets using 5 fold cross validation, in kddcup99 dataset(around 4.5 lakh rows), I am getting maximum of 70-72% accuracy, and one kddcup99 available with 2 lakh dataset, I am getting maximum 85-87% accuracy but the author's have given AUC 1.

For satellite I am getting avg of 65% varying subssmpling size from (1 to 96) and above 70% when subsampling size is more than 500, but they have given 71%(1 to 100).

For breast cancer dataset I am getting maximum of 75% accuracy but they have given AUC 0.97.

I have implemented exactly same but don't know why this. Can someone please help me. I can share my accuracies as well.

Thank you.


texts: requesting password while git push in jupyter notebook

texts: I am trying to push the commited changes from kaggle kernels using jupyter notebook but while pushing the changes git asking me for username and I really don't know how to give username as it seems there is no textbox or anything where I can type my username.


texts: How to interpret confusion matric for Extreme learning machine ? why it has decimal points in it?

texts:
ELM for phishing URL detection.
Dataset is divided into 70/30. 30% is for testing.
Dataset size: 11,000 Entries.


texts: CV(Curriculum vitae) Recommendation System guidance

texts:
I am building a recommender system which matches people's CV with a vacancy.

So far, I used TF-IDF & Cosine Similarity to get a matching score between a vacancy and a candidate's CV.

I want to know whether there are any other approaches to create such a recommendation engine?


texts: Why don't get the expected result using a SVM training model?

texts: I want to learn a model for recognizing facial emotions. . I used a dataset with 213 samples. I extract firstly features using the Gabor filter. Then, I reduce the data dimensionality with the PCA and the genetic algorithm. Finally, I test performance using cross validation with a test size of 25%, and I got 98% acuraccy. The problem is when I test the SVM model on other images, I don't get the expected result.


texts: Replacing missing value by class conditional mean

texts:
I have two classes, $p(x|y=0)$ and $p(x|y=1)$ with ${{\mu }_{0}}$ and ${{\mu }_{1}}$ as mean and shared covariance matrix $\Sigma $. Now, I have a missing feature ${{x}_{n}}$ for a particular observation. We replace this missing value by the class conditional mean $E({{x}_{n}}|y=0)$ and $E({{x}_{n}}|y=1)$. How do I justify this missing value treatment.

This is actually an exercise. But, something that I noticed is that if $p(x|y=0)$ and $p(x|y=1)$ have means ${{\mu }_{0}}$ and ${{\mu }_{1}}$, then how is their covariance matrix the same?

Assuming their covariance matrix is same, the only advantage I see is that if I take class conditional mean as missing value then in multivariate Gaussian, the effect of this is that the $n^{th}$ row of $\left( x-\mu  \right)$ vector will be $0$ since $\left( {{x}_{n}}-{{\mu }_{n}} \right)$ will be $0$, thus making no impact in calculating $p(y=0|x)$ and $p(y=1|x)$.

Is this explanation correct?


texts: What GPU size do I need to fine tune BERT base cased?

texts: I want to fine tune BERT Multilingual but I'm not aware about the GPU requirements to train BERT Multilingual. I have GTX 1050ti 4GB on my local machine. I want to know what size of GPU is needed and what type of GPU is needed to train BERT. I have access to server resources. Could anyone tell me what size of GPU should I request for on server.


texts: how to sort a multi-level pandas data-frame by a particular column?

texts:
I want to sort a multi-index pandas dataframe by a column but don't want the entire dataframe to be sorted at once. But rather want to sort by one of the indices. Here is an example of what I mean:
Below is an example of a multi-index dataframe.

first  second
bar    one       0.361041
       two       0.476720
baz    one       0.565781
       two       0.848519
foo    one       0.405524
       two       0.882497
qux    one       0.488229
       two       0.303862

What I want to do is to get the following:

first  second
bar    one       0.476720
       two       0.361041
baz    one       0.848519
       two       0.565781
foo    one       0.882497
       two       0.405524
qux    one       0.488229
       two       0.303862

These are generated by hand to show what I want. Notice that the second dataframe is not completely sorted. But within each multi-index, it is sorted in descending order. I have a large dataframe. Is there a simpler way of do it (such as a function) instead of grouping the dataframe based on the indices, and then concatenate the individually sorted dataframes.

Thanks for your help.


texts: Are Correlithm Objects used for anything in the industry?

texts:
This refers to a system described in a book by Nick Lawrence titled "Correlithm Object Technology". The author coined the term "correlithm" as a combination of "correlation" and "algorithm".

The Correlithm Objects ("COs") described in the book are, in the author's view, "primary data tokens" in biological neural systems and are central to "all high-level data representation, storage, and manipulation". In addition to this the author describes the COs as "important mathematical objects of the statistics of bounded, high-dimensional spaces".

Considering these descriptions it seems like these COs could be useful for AI. What I was curious about is whether they are actually used anywhere in the industry, and if so, in what kind of situations?


texts: Anomaly Detection In Univariate Time Series Data Using ARIMA In Python With Updating

texts:
I have trained an ARIMA model on some 15 minute incremented time series data by using the statsmodels library. I would like to determine how anomalous the next 15 minute increment's data I observe is. I would then like to update the model with that data.

I can predict the range of future outcomes within an accepted error margin like so:

model = arima_model.ARIMA(data.COUNTS,(p,d,q)).fit()
print model.forecast(alpha=.001)[2]

That outputs:

  array([[ 16.13395152,  48.47024783]]))

So you could say any number outside that range is anomalous. However, if my next observation were, for example, 50 that would presumably be less anomalous than an observation of 51. Which way do you recommend I use to determine exactly how anomalous an observation is using an ARIMA model?

Also, how should I go about updating my model with the new observation?


texts: How to add Earlystopper in Classifier Model

texts:
I have designed the following Binary Classifier Neural Network Model for a task. I want to add an early stopper to the model so that the model stops at an epoch where it has stopped learning significantly. How can I do that?
Model
X = badge1_data[['1','2','3','Score']]
y = badge1_data['APR']

#Standardizing the Input Features
scaler = StandardScaler()
X = scaler.fit_transform(X)

#Train Test split
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.3)

#Create Model
model = Sequential()
model.add(Dense(4,
                  input_dim=4,
                  kernel_initializer='normal',
                  activation='relu'))

model.add(Dense(2,
                  kernel_initializer='normal',
                  activation='relu'))

model.add(Dense(1,
                  activation='sigmoid',
                  kernel_initializer='normal'))

#Compile Model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

#Fit Model
model.fit(X_train, y_train, epochs = 5000, validation_split = 0.3, verbose=0, batch_size=256)

#Make predictions and convert to binary value
pred_train = model.predict(X_train)
pred_test = model.predict(X_test)

#ROC AUC Score
print('Train AUC = {:}'.format(roc_auc_score(y_train,pred_train)))
print('Test AUC = {:}'.format(roc_auc_score(y_test,pred_test)))

#Accuracy
print('Train Accuracy = ',accuracy_score(y_train,pred_train.round()))
print('Test Accuracy = ',accuracy_score(y_test,pred_test.round()))


texts:
Let's say I wan't to predict the lifespan of an ad in a listing.

I know a bunch of thing from the ad like:

the title
the price
the location
etc

The target value is the duration of the ad in the listing before it's being removed (item has been sold).

What would be the best approach for engineering the target?

I've tried categorizing the log of the duration, but it's not leveraging the cyclic pattern you can see in the histogram of the lifespan (in hours):

x-axis : lifespan in hours


texts: Do we have to split our dataset into training & testing when using ARIMA model?

texts:
I am working on a project where I predict the total quantities sold at the ITEM/DAY leve. As for the model, I decided to with an ARIMA model (I'm using R). For guidance, I decided to follow the two tutorials below:

The first one here

The second one here

The thing that I am confused about is that, in the second one, they split the data into training and testing and they fit the model on the training set and did the evaluation on the test set (all that makes sense). However, in the first article, they didn't do the splitting.

Can anyone guide me through which is the correct approach and provide me with an explanation?

Thank you.


texts: How to decide if gradients are vanishing?

texts: I am trying to debug a neural network. I am seeing gradients close to zero. How can I decide whether these gradients are vanishing or not? Is there some threshold to decide on vanishing gradient by looking at the values? I am getting values close to 4 decimal places(0.0001) and in some cases close to 5 decimal places (0.00001). The network seems not to be learning since the histogram of weight is also quite similar in all epochs. I am using RELU activation and Adam optimizer. What could be the reason for the vanishing gradient in case of RELU activation? Is possible please point me to some resources that might be helpful. Thanks in advance.


texts: stacking features vs concatenating layers

texts:
I am trying to get to the logical intuition of differences between stacking multiple features and passing it via a final block (which could comprise multiple layers and lets say a final classification layer) vis-a-vis passing these features via different layers and concatenate them before passing it via a final classification layer.

To give you a concrete example, if I am classifying images of different types,

I might pass them through a pre trained model(weights) and get a set of features, extract other features of image like maybe some blobs, histogram etc , ensure they have the same dimensionality and stack them and pass them through 1 model
OR i might pass the extracted features through 1 FC layer and then , say, the histogram features through another FC layer and concatenate them and finally pass this concatened output through a final FC layer to get the classification.

i would presume, 2 diff models + concatenation should be applied when the features are co-related but have different characteristics. For e.g. stock prices AND monthly employment numbers BUT if they are coming from the same image, they should be stacked ?? quite confused and any pointers would be manna from the heavens


texts: Explainable AI, how did the computer classified Text?

texts:
My question is not about explaining the model or the algorithm
like which neurons were triggered and what are the parameters of perceptrons.
I will explain further
The problem
I have medical reports I want to analyse and extract diagnoses from it.
I built deep neural network model and extracted diagnoses with good accuracy 80%
That is good.
Now When patient look at the results and say: hmmmm, your results says I have Corona, how did the computer know?
at this moment there is no way to answer that.
The predictive model gives us the diagnoses not how it was diagnosed.
Requirements
Is there any mechanism or technique that allows us to find how did the computer made this decision?
If patient was classified with diabities, the model shoud identify words that led to the identification.
Results : Diabities

Explanation :
               the word "Thirst"
               the word "urinating frequently"
               "weight loss"
               "Fatigue"
               "Insulin"

I have been googling with not much luck and I don't where to look and what is the topic name?
Any idea how to do that or at least guidance of where I can find the answer?


texts: Text Classification on a very small data set with a lot of classes

texts:
I have a data set consisting of 455 rows spread over 21 different classes. The data set is imbalanced as well as you can see below.

job_alerts                    45
howto_apply                   40
application_status            31
salary                        30
job_close_date                30
multiple_role                 27
feedback                      26
assessment_campatilibility    24
interview_reschedule          23
disability                    22
reinstate_application         19
job_account_issue             19
assessment_link_problem       16
age_limit                     16
assessment_timebox            16
cv_past_experience            15
late_for_interview            13
interview_response_time       13
work_experience               11
assessment_validity           10
special_needs_at_work          9

I'm trying to perform classification on this. Firstly, when preprocessing the data I convert the text to lowercase, remove stopwords and punctuation and convert words to their lemmas using spaCy.

So far, I have tried the TextCategorizer in spaCy which uses a simple CNN for classification and I'm getting around 70% accuracy on an 80-20 split of the data. I have also tried various Sklearn algorithms such as SVC, LogisticRegression, RandomForestClassifier and MultinomialDB with the  TfIDVectorizer. I have used GridSearchCV to tune the parameters. The best results I get are with SVC at around 74-75% accuracy.

I want to know what else I can try to improve my accuracy. I am a beginner to NLP and I haven't worked on something like this before. Right now the ideas I have are to improve the TfIdVectorizer through some parameter tuning and to try a One vs all classifier for SVC. What else can you suggest to me?


texts:
My question is what are potential reasons for Naive Bayes to perform well on a train set but poorly on a test set?

I am working with a variation of the 20news dataset. The dataset has documents, which are represented as "bag of words" with no metadata. My goal is to classify each document into 1 of 20 labels. My error rate on training data is 20%, but my error rate on test data is 90% (for comparison, guessing yields 95% error rate). For some reason, my classifier is predicting class 16 for almost all documents in test set. In train set this problem does not occur. Furthermore, this issue persists with different train/test splits. I'm trying to figure out what I'm doing wrong.

Here are some things I've considered:

Is Naive Bayes overfitting to the training set? If Naive Bayes is implemented correctly, I don't think it should be overfitting like this on a task that it's considered appropriate for (text classification).
Is my train/test split bad? I've tried splitting the data in different ways, but it does not seem to make a difference. Right now I'm splitting the data by placing a random 90% sample of documents into the train set and the rest into the test set - separately for each label.
Problems with numerical accuracy? I've implemented the calculations in log probabilities, but in any case, I would expect that problem to manifest in train set as well.
Problems with unseen words? The fraction of unseen words relative to a particular newsgroup is the same, 20%, for both train and test sets.
Problems with Laplace smoothing? I wasn't sure what was the appropriate way to implement Laplace smoothing for this task, so I tried a variety of ways. To my surprise they all yielded very similar results.
Imbalanced dataset? Doesn't look like it. There are approximately as many unique words inside documents of label 16 as there are in other classes. Also the number of documents per each label is roughly even.

Edit: Turns out I had an implementation bug. I won't detail it here as it will be unlikely to help anyone with similar issues.


texts: Flow_from_directory() won't load any image

texts:
i'm trying to load images for training a CNN using keras flow_from_directorY(), but seems that is having hard times finding the images. i'm running the code on google colab

!unzip "/content/drive/My Drive/fruits.zip" -d "/content/drive/My Drive"

I used this command to unzip my dataset and then if i try to visualize how many images i have for every folder, it seems that it recognises the images

for folder in os.listdir(os.path.join(path,'training')):
folder_count = len(os.listdir(os.path.join(path,'training',folder)))
image_count.append(folder_count)
class_names.append(folder)
print('{:20s}'.format(folder), end='')
print(folder_count)
print('-'*24)
print('Number of Classes:', len(class_names))
print('Average number of images per Class: ',
np.array(image_count).mean())

however, if i try to use flow_from_directory with those paths, i got that it has found 0 images

train_batches = train_datagenerator.flow_from_directory('/content/drive/My
Drive/fruits-360_dataset/data/training', target_size=(224, 224),
color_mode="rgb", classes="categorical" ,  batch_size=32, subset='training')
val_batches = train_datagenerator.flow_from_directory('/content/drive/My
Drive/fruits-360_dataset/data/training', target_size=(224, 224),
color_mode="rgb", classes="categorical" ,  batch_size=32,
subset='validation')
test_batches = test_datagenerator.flow_from_directory('/content/drive/My
Drive/fruits-360_dataset/data/test', target_size=(224, 224),
color_mode="rgb", classes="categorical" ,  batch_size=32, shuffle=False)

Found 0 images belonging to 11 classes.
Found 0 images belonging to 11 classes.
Found 0 images belonging to 11 classes.

if someone could help would be really appreciated! :)


texts: 2) Create data frame using following object

texts:
Create data frame using following objects.
workshop <- c("R", " SPSS ", NA , " SPSS ", " STATA ", " SPSS ")
gender <- factor (c(" Female ", " Male ", NA , " Female ", " Female ", " Female ") )
q1 <- c(4 , 3 , 3 , 5 , 4 , 5)
q2 <- c(3 , 4 , 2 , 4 , 4 , 4)
q3 <- c(4 , 3 , NA , 5 , 3, 3)
q4 <- c(5 , 4 , 3 , 3 , 4 , 5)

a)  Create a dataframe consisting of only the first two columns.
b)  Create a dataframe consisting of only the first and last row.
c)  Create a dataframe called df2 where every entry in the q3 and q4 columns is 0.
d)  Sort df by gender.


texts: Build a relevancy scoring model of articles using NLP

texts:
I'm really new to Data Science and text mining. I want to build a relevancy scoring model. Suppose I have a bag of words (guns, military, terrorists). I also have a list of articles. I want to find if the articles are relevant to the bag of words.

What I have thought of doing:

clean the article by removing stop-words, punctuation and stemming.
Try to find the average score of each word of the article with the
bag of words using wordnet.wup_similarity
Then compare the
scoring of the articles. If they pass a threshold, they will be
considered relevant, else not.

I have also seen TF-IDF and topic modelling and then I can find the semantic similarity between the topics identified and the bag of words and see if relevant.

But this might be  a very naive way of doing it. Can anyone suggest any methods/papers/algorithms to perform the same?

Any help is appreciated. Thanks in advance.


texts: How to find transictions and group variables

texts:
Having a dataset like this:

structure(list(id = c(1L, 1L, 1L, 1L, 2L, 2L, 2L), stock = c("stockA",
    "stockB", "stockC", "stockD", "stockA", "stockB", "stockD"),
        var1_before = c(-0.3957, -0.0201, -0.3957, -0.3729, 0.0498000000000001,
        -0.5075, -0.3242), var2_before = c(-1.3106, -1.4492, -1.3106,
        -1.6134, -1.3222, -1.5452, -1.3168), var3_before = c(3.3374,
        2.6408, 3.3374, 3.4728, -0.2173, 3.6311, 3.0884), var4_before = c(-1533,
        -1.3378, -1533, -1.5256, -1.6596, -1.7272, -1.4142), var5_before = c(0.3841,
        0.1647, 0.3841, 551, 3.5372, 0.3317, 0.4339), var1_after = c(-0.4975,
        -0.4107, -0.3557, -0.5223, -0.2173, -0.2003, -0.4473), var2_after = c(-1.6707,
        -1.5982, -1.4963, -1.6512, -1.6596, -1.7075, -1.6361), var3_after = c(3.9367,
        3.7744, 3398, 3.9537, 3.5372, 3.4673, 3.7018), var4_after = c(-1.6377,
        -1.5513, -1.6543, -1.6823, -1.5497, -1.3507, -1.8195), var5_after = c(0.6483,
        0.5484, 0.4024, 0.3634, 0.4352, 0.3594, 0.3441)), class = "data.frame", row.names = c(NA,
    -7L))

the variables are
id: is a user
stock: the name of a stock that a user (id) has an impression/sentiment for this. For every user the stock is unique
var1_before-var5_before and var1_before-var1_after: are variables which are score of sentiments. For example var1_before is the sentiment the user has before a specific event and var1_after is the sentiment score the user has after a specific event. the same for 2,3,4,5.

I know that there are some user which can before and after event and moved from var1_before to var3_after.

For every stock how is it possible to find how most of users were in before label, for example for stockA var1_before and var4_before affect most but for stockB var2_before seems to exist most.

Is there any machine learning approach which could make it?


texts: Multi-task learning, finding a loss function that "ignores" certain samples

texts:
I'm training a convolutional neural network with 1 input, and 3 outputs: a classification and 2 regression outputs.

I initially used 3 separate models, but I found that combining the 1st and 2nd loss functions into 1 model improves accuracy/rmse for both. I'm applying multi task learning.

Now I'm experimenting with incorporating the 3rd loss function into the same model with the first 2.

My challenge is that the 3rd loss function is only valid when the 1st classifier identifies a positive sample (it's a regression output that measures the identified sample). These labels are known at training time.

I attempted:

Standardizing the labels and impute a 0 value for the 3rd regression for all negative samples. That approach biases towards 0 in some cases, so I threw it out.
Alternate training between all samples on the 1st and 2nd loss, and only positive samples using the sum of all 3 loss functions. Mixed results here, but the 3rd loss function seems to perform worse than it does in its own separate model.

Question:

I'm considering how I might re-structure the loss function (currently L2 loss) so that it treats a 0-value as irrelevant (e.g. no gradient is propagated when the label is 0).

Any thoughts?


texts:
I have experienced that most of the datasets contain missing values, which make our task bit challenging.

Please let me know how to fill up those missing values in an efficient way? and is there any specific techniques to handle missing values?


texts: Understanding Exclusive-OR predictions in Elman network

texts:
I have been reading Elman network paper, which can be found Here. in page 185, under Exclusive-OR section it was written as follows.

  Notice that, given the temporal structure of this sequence, it is only sometimes
  possible to predict the next item correctly. When the network has
  received the first bit-1 in the example above-there is a 50% chance that
  the next bit will be a 1 (or a 0). When the network receives the second bit (0),
  however, it should then be possible to predict that the third will be the XOR,
  1. When the fourth bit is presented, the fifth is not predictable. But from
  the fifth bit, the sixth can be predicted, and so on.

So, to give a context, author was explaining how we can use networks with memory to form a XOR Gate.

What i don't understand here is this sentence

  "When the network receives the second bit (0),however, it should then
  be possible to predict that the third will be the XOR,1"

How can we be sure that the third element is 1, given second element is 0. And again, why can't we predict 4th element.


texts: Reconstructing input image from layers of a CNN

texts:
I've been trying to implement neural style transfer as described in this paper here According to the paper,

  we can visualise the information at different processing stages in the
  CNN by reconstructing the input image from only knowing the network’s
  responses in a particular layer.

My question is, how exactly does one go about reconstructing image from a single layer?
I'm implementing this in pytorch. I've the output from layer conv4_2 stored in a tensor of shape [1,512,50,50] but how do I visualize this?

Here's a part of my code, if that helps.

vgg = models.vgg19(pretrained=True).features

for param in vgg.parameters():
    param.requires_grad_(False)

device = torch.device("cpu")

vgg.to(device)

content_img = Image.open("image3.jpg").convert('RGB')
style_img   = Image.open("image5.jpg").convert('RGB')
content_img = transformation(content_img).to(device)
style_img   = transformation(style_img).to(device)

def get_features(image, model):

    layers = {'0': 'conv1_1', '5': 'conv2_1',  '10': 'conv3_1',
              '19': 'conv4_1', '21': 'conv4_2', '28': 'conv5_1'}
    x = image
    features = {}

    for name, layer in model._modules.items():
        x = layer(x)

        if name in layers:
            features[layers[name]] = x

    return features

content_img_features = get_features(content_img, vgg)
style_img_features   = get_features(style_img, vgg)

target_content = content_img_features['conv4_2']

How do I reconstruct the image from the output of conv4_2?


texts: How to efficiently remove low frequency words from a large corpus with python

texts:
I have a corpus with about 135k documents (several pages per document). And a vocab of about 800k. I noticed that something like half of my vocab is words with a frequency of 1 or 2, so I want to remove them.
So I'm running something like this:
remove_indices = np.array(index_df[index_df['frequency'] <= 2]['index']).astype(int)

for file_name in tqdm(corpus):
    content = corpus[file_name].astype(int)
    content = [index for index in content if index not in remove_indices]
    corpus[file_name] = np.array(content).astype(np.uint32)

Where corpus looks something like:
{
    'filename1.txt': np.array([43, 177718, 3817, ...., 28181]).astype(np.uint32),
    'filename2.txt': ....
}

and each word was previously encoded to a positive integer index.
The problem lies in content = [index for index in content if index not in remove_indices] which needs to go through len(remove_indices) * len(content) number of checks with each iteration. This would take forever. Any tips on how to speed this up?


texts: Stationary time series for clustering algorithms

texts:
I have a set of time series data that I would like to feed into a clustering algorithm (like k-means, using dynamic time warping as the distance function). After standardizing the data with mean 0 and variance 1, the k-means classifier generated a batch of centroids that seemed to fit the data pretty well.

The only question I have is whether the data should be stationary. Models such as ARIMA require for the data to be stationary due to the nature of it. However, the data I want to cluster is mortgage rates as a function of time, which could be subject to seasonal trends, which could be useful when clustering other future time series data.

The question is: do clustering algorithms for time series data generally require for the data to be stationary?


texts: How Xgboost or Decision trees deal with discontinous data in tabular form for time series

texts:
I am participating in a kaggle competition by the name "predict future sales". There the objective is to predict sales for each shop for each item for next month. The data is given such that only those those items are mentioned when some item was sold. For example item with id "1" was sold on month january so its record will be given in data .

I had made combination of items that were sold in a given month with all shops . So if two items with id "1" and "2" are sold in january there will be 56*2 total combinations in that month and if in another month three items with id "4","5","1" are sold then there will be 3*56 total combinations. I have done this because in test set we have to predict all items for all shop combinations. What i am saying is even if item with id "1" is sold in shop 1 only i will make combination for all shops and set zero for all shops where it was not sold.

The problem is when i convert it in to tabular form time series are discontinous, beacause if an item is not sold at all in some months there will be no combination or no record. My concern is can XGboost itself know that certain item is not sold in that month? Will it pick seasonality? Should i convert my data in to time series format and then convert time series in to tabular form. This will make data too much large 22000(items)*shops(56)*24(months) around 2.9Million records. I want to know how Xgboost will determine what happen to that item in particular month if there is no record for that item in certain months. Say for example sweaters were sold in january and then they were again sold on next years january only.

Any help regarding framing of this problem will be appreciated.


texts: Relu does have 0 gradient by definition, then why gradient vanish is not a problem for x < 0?

texts:
By definition, Relu is max(0,f(x)). Then its gradient is defined as:
1 if x > 0 and 0 if x < 0.

Wouldn't this mean the gradient is always 0 (vanishes) when x < 0? Then why do we say Relu doesn't suffer from the gradient vanish problem?


texts: Mapping between original feature space and an interpretable feature space

texts:
I'm reading the following really interesting paper

https://arxiv.org/pdf/1602.04938.pdf on local interpretable model explanations

on page 3 however particularly section 3.3 Sampling for Local Exploration they mention obtaining perturbed samples $z' \in \{0,1\}^{d'}$, it then says

"we recover the sample in the original representation
$z \in \mathbb{R}^{d}$
and obtain $f(z)$ "

with no indication how this is done, surely the map is not injective? If not how would you know you recovered the correct sample? To this end, i wondering how something like this might be done in practice, moving from one feature space $\mathbb{R}^{d}$ to another $\{0,1\}^{d'}$. I'd really appreciate any help.


texts:
Trying to locate the names of countries in my data.frame where the most spoken language is not an official language. My columns data.frame is World. The column country.y is the name column of all the countries. Column isOfficial has data in it that is 't' or 'false' representing that the official language is 'true' or 'false'. Another column is percentage representing what percentage of a language is spoken.  So far this is what I came up with.

subset(World$name.y, World$isOfficial != 't' & World$percentage > 50)


texts: In the context of Deep Learning, what is training warmup steps

texts: I found this term "training warmup steps" in some of the papers, what exactly does this term mean? Has it got anything to do with "learning rate"? If so, how does it affect?


texts: What ML/DL techniques power Youtube/Netflix search systems?

texts:
Video platforms like YouTube, Netflix, Amazon prime have an excellent search system - given a search string, find most relevant videos. Which Machine Learning /Deep Learning techniques used for this?

Any pointers will be of great help


texts: What's the best metric for evaluate an estimator for a multi class problem with class imbalance dataset?

texts:
accuracy, precision, f1, ROC are good for binary single class problem.

but for more complex problem (imbalance multi-class problem), what should i use?

Do you have any recommendation?


texts:
I have a large number of x86_64 cores available to me, but no GPUs or TPUs. Which Machine Learning techniques are suited for execution on a CPU? I would imagine more "statistical learning" techniques rather than "deep learning".

The reason I ask is that I'm interesting in studying some state-of-the art techniques, but I'm in the situation noted above.


texts: Time series classification with convolutional neural network

texts:
I have to classify a time series signal with a CNN (not a LSTM or some kind of RNN). The input signal is a window with a width of 90 samples (currently, but can be changed) and 59 sensor values. So my signal can be represented by a 90x59 matrix for a single class. I have 28 classes.

Now I´m looking for a good solution to classify this. My first try ends in a very poor result of 24% accuracy and I´m thinking of multi channel models for all 59 channels in my model.

Is there any good way to start for some kind of good prediction results?


texts: What are key dataset requirements for topic models and word embeddings?

texts:
I have a dataset of 2000 documents where avg doc size is 300 words. The vocab is dominated by domain-specific words.

My goal is to find similar documents. For this, I tried LDA, LSI, Doc2Vec (topics=100) but results are not great. LSI is better than the others in my dataset. I also tried word2vec (size=100) & word movers distance but again no luck. I am thinking of trying POS tags & then building some ontology model but not sure.

Are my results poor because of a bad dataset? What are some other techniques that I can try?


texts: Input and output feature shapes in CNN for speech recognition

texts:
I am currently studying this paper and are trying to understand what exactly the input and output shape is.  The paper describes an acoustic model consisting of using cnn-hmm as the acoustic model.  The input is a image of mel-log filter energies visualised as spectograms. The paper describes a method  for phone recognition in which (as far I understand) applying a CNN on these spectograms with a limited weight sharing scheme should be beneficial for phone recognition.

The input shape as far i understand, is 9-15 frames, which seem a bit confusing, as they don't consider number of phonemes a utterance may have, or the length of them, but simply just "choose" a number of frames to operate with.. The number doesn't seem to be connected with the output in any way - or am I misinterpreting something?

For the output

  We used 183 target class labels, i.e., 3 states for each HMM of 61 phones. After decoding, the original 61 phone classes were mapped to a set of 39 classes as in [47] for final scoring. In our experiments, a bigram language model over phones, estimated from the training set, was used in decoding. To prepare the ANN targets, a mono-phone HMM model was trained on the training data set, and it was used to generate state-level labels based on forced alignment.

So the output is divided in to 183 classes, being mapped into HMM's with 3 states for each 61 phonemes, and the ANN target (As I see it target = posterior probability) by training a monophone hmm with forced alignment. I am not sure I understand this process.  If the ANN targets are those the CNN should aim/regress to and at the end classify the state based on, Why then process the input?.. why not make a simple DNN that does the regression/classification?

It looks like the improvement lies in the use of forced alignment here, and only on monophone? where is the improvement?

And again how I am supposed to link the input shape and the output shape based this? This would require the audio files to have a certain length, the length of the audio is never specified, so I am assuming that this is not the case.