texts: Difference between KNN-DTW and DTW Template matching?

texts: Is there a distinction between kNN-DTW and DTW template matching? Essentially DTW template matching computes the DTW for each point in the training set, similar to the KNN-DTW algorithm. Is there a difference between the two methods?


texts:
I'm new to the data science world, and I hope to solve a problem using deep learning methods, I started learning how FNN and CNN work and when I saw how many models and methods are the I got a bit confused and decided to ask for your help in pointing me at the right direction,

The problem I need to solve: I have a data set which unlabeled and consist of actions taken by a multi-agent system, that will probably contain

Time | Role of Agent | Action taken | Position of agent | outcome ....

I need to train a model\models to detect a strategic sequences of actions (i.e rows in the data) that means that I need a model that will take into consideration not only the current row in data but also the rows that came before (means actions taken by other agents) and if possible rows that will be inputted later.

the output needs to be a classification on the data into strategic plays, such that a play is a few actions that happend one after the other,

the dataset is still unprepared and we are just in the planning and learning stage, by I hope that ow that you understand the problem that I'm facing a bit you can help me decide on a Deep Learning model(or models) that can meet the requirements and preform good for such task, so I can really dig into that model and learn plus if you know of any articles that can be of assistance it will be great as well

Thanks for the help!


texts: I've been playing around with some data, and I've created a decently large similarity matrix (each column value for a row represents how similar the document corresponding to the column is to the document corresponding to the row) and I'm trying to scale this data down to 2 dimensions so I can visualize it on a scatter plot, where each point represents a document, and the closer two documents are, the closer they would be on the scatter plot. I've tried using Sammon Mapping to accomplish this, but the problem is it takes a significant amount of time to run. Any ideas?


texts: Is there a common strategy to measure if a difference-significance of two areas under two ROC curves

texts:
I conduct sound detection experiments with mice.
I have a stimulus sound and a "noise" sound that shoukd be ignored.

I want to measure how well the mouse ignors the noise (with respect to, say, ignoring 100% of the noise stimuli).

I have sessions with, say 200 trials of stimulus sound that sould be detected and 40 trials of noise sounds that should be ignored.

I created the confusion matrix and ROC curve (I use Matlab).
Now I want to know how well my mouse is performing compared with "ignoring all noise stims".

Is there a way, or a common used formula, to get a evaluation or a confidence interval of "how good is my ROC AUC comparing to AUC(ROC) = 1"?

Thanks!


texts: How append works in hdfs? Where the newly created instance of file is placed?

texts:
I'm a Newbie to Hadoop!

By web search and after going through hadoop guides, it is clear that hdfs doesn't allow us to edit the file but to append some data to existing file, it will be creating a new instance temporarily and append the new data to it.

That being said,

1.would like to know whether the new file or temp file is created in the same block or in a different block?

2.What will happen if the revised file exceeds the previous  allocated block size?

Any help would be greatly appreciated!


texts: Preparing training data for NLP machine learning task

texts:
I have the natural language sentences as follows:
This is a black chair. It is next to the table.
Each phrase that represents an object is annotated with an object Id. For example, in the above sentence, we have:
This: 15, black chair: 15, It: 15, table: 14
(where, 14 and 15 are object Ids)
I would like to train a model to predict the object Id of each phrase representing an object for a new sentence. From what I understand, each training example will consist of the following structure:
Inputs: sentence + object phrase
Output: object id (from 18 available ids)
I would need to repeat the above for each object phrase in a sentence
My question: How do I prepare the training data for this task? How do I represent each object phrase (eg: 'black chair') and each sentence for training the neural network?


texts: Matlab Optimization. Meaning of warning: "The slope should be 2. It appears to be 1."

texts:
I'm using the manopt package to solve some optimization problems in matlab. The problem is of the form.

  problem.cost = @(x) f(x)
  problem.egrad = @(x) g(x)

After the problem definition, I check the gradient consistency using the following call:

  checkgradient(problem)

And then I get the following message:

  "The slope should be 2. It appears to be: 1. If it is far from 2, then
  directional derivatives might be erroneous. The residual should be 0,
  or very close. Residual: 5.36216e-16. If it is far from 0, then the
  gradient is not in the tangent space."

Since the algorithm is performing very badly even on very simple examples, I suppose that there is some mistake in the implementation of the gradient. I have some questions about how to interpret the message from the consistency checker.

What is special about the value 2 for the slope? I have found several other codes on the internet where when executing the gradient consistency checker we get the messages of the form: "The slope should be 2. It appears to be 2.00001" or "The slope should be 2. It appears to be 1.9998". Additionally it looks like the value 2 is a specified a priory in the code of the manopt package. See for instance this code.
When the slope is far from 2, as in my case, what does that mean? In my case, I don't know how to interpret this. The function I'm trying to optimize is a multilinear polynomial on real variables. Each monomial has 5 variables. But since each variable appears at most once in each monomial, the partial derivative of each variable x is a multilinear polynomial of degree 4 over the other variables.


texts: Can we apply to GridSearchCV to Logistic regression .?

texts:
When I apply GridSearchCV to my model Logistic Regression, it's continuously throwing below error.
I understand that it's trying to convert string to float. But that's was my data. So how can I convert my x value to float? Can any please hep me with this issue. Thank you.

Error: could not convert string to float: 'great im readi go home'

Sample data

Airline_sentiment     text

0     said

1     plu youv ad commerci experi tacki

2     didnt today must mean need take anoth trip

penalty = ['l1', 'l2']

C = np.logspace(0, 4, 10)

hyperparameters = dict(C=C, penalty=penalty)

gs = **GridSearchCV**(lr,hyperparameters,cv=5)


texts: AUC calculated for a given target class while there are more than 2 groups to compare with

texts:
I have used random forest classifies to classify 4 groups (or class). For each group (target class) the AUC has been generated. Using "Test&Score" (from orange) provides AUC for each target class (Lets say target classes are A, B, C and D) and also average over classes.

How's AUC been generated for a given target class while there are 3 more groups to compare with?

Does the AUC for target class ( let's say "A") is calculated considering other 3 ("B", "C", and "D") groups as one group (one negative group)? Because AUC is calculated for normalized data (0 to 1), assuming 0 as negative and 1 as positive cases.


texts: Fitting model to differenced time series

texts:
I have a time series on daily stock price of company(2013 data points).I took a first order difference and  the following acf and pacf plots of the differenced series were obtained. However, I am unable to come up with a suitable ARMA model. Can anyone please help me with this?!

The above plots don't seem to converge to zero and the acf and pacf values are significant at high lags which I wonder would be appropriate to consider for orders in ARMA model.
The graph given below is that of the original series.


texts: Ethics of using Existing Algorithms in Competitions

texts:
Lets say a paper is published which describes a data science algorithm, and the paper is made available on arxiv (no patent or anything else mentioned in paper). The paper is by a university researcher, and the algorithm is not something well known.

A. Is it considered ethical to use my own implementation of that algorithm for:

work/own profit
own learning (I assume yes)
Competition (with monetary gain).

B. On doing so, is it correct that it is the right thing to do to cite the paper (lets say I am required to hand over competition code).

Now lets say the author published some code to go along with the paper under an MIT/other open source license on github, and I mess around with it to fit my needs. What if the paper was by a company (like facebook or google, but does not cite any patent or limitation on use of the code/method) Do the same rules apply or are there other considerations?

I guess I have not found a good blog covering this and any surrounding issues I may not be aware of as self-taught professional human running into this for the first time.


texts:
I am trying to build a neural net that will overfit random walk path. So, far I wasn't able to get a neural net that we shatter/overfit. I was wondering which parameters I should explore, or which guide lines I should follow to achieve this (wierd) task.
Code:
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD,Adam
from keras import regularizers
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import random
import math

model=Sequential()
num_units=100
model.add(Dense(num_units,input_shape=(1,),activation='tanh'))
model.add(Dense(num_units,activation='tanh'))
model.add(Dense(num_units,activation='tanh'))
model.add(Dense(num_units,activation='tanh'))
model.add(Dense(1, activation='tanh')) #output layer 1 unit
model.compile(Adam(),'mean_squared_error',metrics=['mse'])

num_of_steps=3000;
step_size=0.01
x_all=list(range(0,num_of_steps))

##########################random walk functino
random_walk=[0]
for i in x_all[1:]:
    f=np.random.uniform(0,1,1)
    if f<0.5:
        random_walk.append(random_walk[i-1]+step_size)
    else:
        random_walk.append(random_walk[i-1]-step_size)
########################

x1=list(range(1,1000,2))
x2=list(range(2,1000,2))
y1=[random_walk[x] for x in x1]
y2=[random_walk[x] for x in x2]

model.fit(x1,y1,epochs=2500,verbose=1)
fit1=model.predict(x1)
fit2=model.predict(x2)
plt.plot(x1,y1,'k')
plt.plot(x2, y2, 'r')
plt.scatter(x1, fit1, facecolors='none', edgecolors='g') #plt.plot(x_value,sample,'bo')
plt.scatter(x2, fit2, facecolors='none', edgecolors='b') #plt.plot(x_value,sample,'bo')

this is an example of the results I am getting:

And I would want to get a better fit (to my sample data: x1,y1).


texts: What is complexity of Nearest Neigbor graph calculation and why kd/ball_tree works slower than brute?

texts:
Consider sklearn NearestNeighbors:
nbrs = NearestNeighbors(n_neighbors=2, algorithm=method  ).fit(X) # 'ball_tree'
distances, indices = nbrs.kneighbors(X)

There are several choices for 'algorithm' -
‘ball_tree’ will use BallTree
‘kd_tree’ will use KDTree
‘brute’ will use a brute-force search.
‘auto’ will attempt to decide the most appropriate algorithm based on the values passed to fit method.
Observe a strange thing - "brute" works faster, than tree methods, if the dimension of the space is large enough:

Question 1 Is there any explanation why kd/ball tree work slower and why they are used as "auto" ?
It seems like a bug in sklearn.
Let us look on the speed of kd_tree:

Question 2 what is the theoretical complexity of the kd_tree and ball_tree with respect to sample_size and dimension of the space ? (May be other things - the simulation is done for the uniformly sampled points [0,1]^d).
Remark: it is clear that for "brute" complexity grows quadratically with respect to sample size. As one can see from data above - kd_tree works quite linearly with respect to sample_size.
The code was running on colab.research.google.com
import time
from sklearn.neighbors import NearestNeighbors
import numpy as np
import pandas as pd

dim = 5
n_sample = 10**4

df_stat = pd.DataFrame()
c = 0
t00 = time.time()
for i in range(1): # repeat test several times
  for dim in [5,10,20]:
    for n_sample in [10**5]:
      X = np.random.rand(n_sample, dim)

      for method in ['brute','kd_tree','ball_tree','auto']:
        c += 1
        df_stat.loc[c, 'Method'] = method
        df_stat.loc[c, 'Dim'] = dim
        df_stat.loc[c, 'N_sample'] = n_sample
        t0 = time.time()
        nbrs = NearestNeighbors(n_neighbors=2, algorithm=method  ).fit(X) # 'ball_tree'
        distances, indices = nbrs.kneighbors(X)
        df_stat.loc[c, 'Time'] = time.time()-t0
        if method == 'brute':
          indices_save = indices.copy()
        difr = indices_save - indices
        df_stat.loc[c, 'Coincide with Brute'] = (np.sum( np.abs(difr)) == 0)
        print(df_stat.tail(1))

df_stat

PS
The motivation to ask comes from:
https://cstheory.stackexchange.com/questions/47034/number-of-connected-components-of-a-random-nearest-neighbor-graph/47039?noredirect=1#comment104468_47039


texts: "ValueError: Index contains duplicate entries, cannot reshape" error when I try to use pd.MultiIndex.arrays

texts:
I have data which includes id , gender , collected time test name and Test values , Units of measurement

Test Names will include all tests that a patient taken and Value col will have its corresponding test result.

I want to analysis on only certain tests and retrieve corresponding test values from "value" col . The analysis will be on those tests and their values , so I thought it would be good idea to pivot on those test names and test values. However when I add TS col I get an error and adding any other test name in the multiindex code does not throw an error.

Steps:

 df_s.head(30).dropna()

Here in the below screenshot we can see there multiple test taken for each requisition id:

In the below code Iam only getting tests which I want to do analysis
1#

df_s2 = df_s[df_s['Test'].isin(['TOTAL TRIIODOTHYRONINE (T3)','TOTAL THYROXINE (T4)','FREE THYROID 3','FREE THYROID 4','Human Chorionic Gonadotropin (hCG)','BILRUBIN'])]

2# Resetting the index:

df_s3=df_s2.set_index(['ID', 'Name', 'Age', 'Sex', 'CT', 'RT', 'Test', 'Test_Result', 'Units']).reset_index()

3# applything multiindex

idx = pd.MultiIndex.from_arrays([df_s3['ID'], df_s3['Name'], df_s3['Age'], df_s3['Sex'], df_s3['CT'],df_s3['RT'], df_s3['Units'], df_s3['Test'],  ])
#, df_s3['Unit of Measure']
df_s5 = df_s3.set_index(idx).Test_Result.unstack(fill_value='')
df_s5.columns.name = None
df_s6= df_s5.reset_index()
df_s6.head(100)

Result:

I get this result if do not add TSH (from Test Col)
 Code with TSH test:
Retry 1# with TSH

df_s2 = df_s[df_s['Test'].isin(['TOTAL TRIIODOTHYRONINE (T3)','TOTAL THYROXINE (T4)','THYROID STIMULATING HORMONE (TSH)','FREE THYROID 3','FREE THYROID 4','Human Chorionic Gonadotropin (hCG)','BILRUBIN'])]

df_s3=df_s2.set_index(['ID', 'Name', 'Age', 'Sex', 'CT', 'RT', 'Test', 'Test_Result', 'Units']).reset_index()

idx = pd.MultiIndex.from_arrays([df_s3['ID'], df_s3['Name'], df_s3['Age'], df_s3['Sex'], df_s3['CT'],df_s3['RT'], df_s3['Units'], df_s3['Test'],  ])
#, df_s3['Unit of Measure']
df_s5 = df_s3.set_index(idx).Test_Result.unstack(fill_value='')
df_s5.columns.name = None
df_s6= df_s5.reset_index()
df_s6.head(100)

Question1  (Retry 1# with TSH ): Please help me with the correct approach, what I understand the error is because once it convert it is not finding any unique index but not sure how to resolve it.

Question2: When I proceeded to go ahead without tsh, after conversion of test col- rows to cols, I get blank values in respective test col ( example T4 col) because 1) the person has taken the test but there is no value in the dataset(python is treating it as Null value and can be imputed/rejected - no issue 2) the patient has not taken this test but has taken atleast one other tests may be T3, hcg etc but not this test- this is considered as string '' . I want to get rid of these rows for amy analysis .. is there an approach while transforming the data to take care of so that I only want the result of the code to have T4 and its value( numeric or null). I do not want a scenario where the person has not taken test at all. OR is there a way to impute these values so I will know the person has taken T4, T3 but not Hcg , bilrubin etc?

Please advise. Long questions but I hope it this explanatory


texts: Error-analysis and evaluation of a model using Python?

texts:
My method of evaluating a model is the following :

Split the training data set and do cross validation to obtain an accuracy of my model on my cross validation data set.
Use the parameters that gave me the best accuracy and use predict() on my test data set ( hold-out data set )
Run a little 'for' loop to check how many labels i missclassified ( Let's assume i'm doing classification ) on my test dataset for which i hid the real labels.
I'd look at the percentages of 'accuracy' given by each algorithm and pick the best accuracy.

My question :

What can be done to improve my method of error-analysis and model evaluation using Python? Code snippets and their purposes would be helpful.

Thanks!


texts: What are the rules when extracting SVO triples from preprocessed text?

texts: If you have some already preprocessed text that is tagged, what are the rules to extract Subject-Verb-Object (SVO) triples if you want a triple like (word, word, word). Can you give the sentence as example and extract all triples? Do you just need to find all combinations without repetition from set of N words?


texts: Dynamic Time Warping Linear Algebra and Pseudocode Explanation

texts:
Can someone explain some questions I have about the pseudocode in the Wikipedia article for Dynamic Time Warping? (I'm talking about the implementation with the window constraint parameter w.

On the line DTW[0, 0] := 0, why are we setting the first element to zero after setting the entire matrix to infinity? I'm sure there's some reason, but I can't figure out why.

On the lines initializing the second set of for loops:

for i := 1 to n
    for j := max(1, i-w) to min(m, i+w)

why do we skip the first row/column? I get the bounding idea in the inner loop with the window constraint, but from what I understand, in DTW, not using the first column and first row is essential. Why?

Within the for loops:

cost := d(s[i], t[j])
DTW[i, j] := cost + minimum(DTW[i-1, j  ],    // insertion
                            DTW[i  , j-1],    // deletion
                            DTW[i-1, j-1])    // match

I understand the cost is the current distance between two elements in the respective time series, but what are we doing when we set DTW[i, j] to

cost + minimum(DTW[i-1, j  ],    // insertion
               DTW[i  , j-1],    // deletion
               DTW[i-1, j-1])    // match

I'm not the best at Dynamic Programming, so I don't understand this line as well. Why are each cases considered an insertion, deletion, and a match?

Finally, for the last line:

return DTW[n, m]

Why are we returning the last element? I thought DTW was supposed to return the aligned time series which I figured would be represented by the warping path in the matrix.


texts: where can i find the algorithm of these papers?

texts:
I am reading about clinical NER
I found 2 papers talking about it
Paper 1
and
Paper 2
They are talking about algorithms and ML has been used to approach clinical NER.
I could not find anywhere on how exactly these algorithms are implemented in these papers.
Can anyone help me find it or build it?


texts: Output probabilities for each class on tensorflow sigmoid function

texts:
I have a piece of code that uses tf.nn.softmax to predict whether does a image belongs to either class 0, 1, 2... etc.

However, I want to edit the code to using sigmoid as the activation function and outputing all the probabilities, and setting those with probabilities >0.5 as one of the classes identified in the image.

This is the code which I am trying to run: https://github.com/satyenrajpal/Concrete-Crack-Detection

I believe this is the code snippet where my edit should be made:

            for counter,image in enumerate(test_images):
                #break up images into 128*128
                broken_image,h,w,h_no,w_no = break_image(image,128)

                output_image = np.zeros((h_no*128,w_no*128,3),dtype = np.uint8)

                feed_dict = {x: broken_image}
                batch_predictions = sess.run(predictions, feed_dict = feed_dict)

                print("here is one loop")

                print(batch_predictions)

#                file = open("test.txt","w")
#                bpred_str = batch_predictions.astype('str')
#                file.write(bpred_str)
#                file.write(" ")
#                file.close()

                results=np.concatenate((results, batch_predictions))

                matrix_pred = batch_predictions.reshape((h_no,w_no))
                #Concentrate after this for post processing
                for i in range(0,h_no):
                    for j in range(0,w_no):
                        a = matrix_pred[i,j]
                        output_image[128*i:128*(i+1),128*j:128*(j+1),:] = 1-a

                cropped_image = image[0:h_no*128,0:w_no*128,:]
                pred_image = np.multiply(output_image,cropped_image)

I tried to print batch_predictions, but instead it prints out something like:

[1 1 0 1 1 0]

Another snippet is:

#Predict the class
y_pred = tf.nn.sigmoid(layer_fc2)
print("a: %s",y_pred)
#print("Class Probability: %s"%(sess.run(y_pred)))
self.y_pred_cls = tf.argmax(y_pred, dimension=1,name="predictions")

#Cost Function
cross_entropy = tf.nn.sigmoid_cross_entropy_with_logits(logits=layer_fc2, labels=self.y_true)
print("b: %s",cross_entropy)
cost = tf.reduce_mean(cross_entropy)
optimizer = tf.train.AdamOptimizer(learning_rate=1e-4).minimize(cost)

#Predict
correct_prediction = tf.equal(self.y_pred_cls, self.y_true_cls)
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
probabilities=tf.nn.sigmoid(y_pred)
print("Class Probabilities: %s", probabilities)
return optimizer, accuracy

I also tried to print the output of tf.nn.sigmoid(y_pred), but it gave:

Class Probabilities: %s Tensor("Sigmoid_1:0", shape=(?, 2), dtype=float32)

I need help on how to print out the individual probabilities of the classes when I run the model on a unlabelled data. Thank you in advance!


texts: Machine learning or NLP approach to convert string about month ,year into dates

texts:
I'm currently in the process of developing a program with the capability of converting human style of representing year into actual dates.
Example : last year last month into December 2018
string may be complete sentence like : what were you doing 5 years ago

it will gives 2014

The purpose is to evalute human style of represting year or date into actual date, i have created collection of this type of strings and matching them with regex.

I have read some machine learning but I'm not sure which algorithm suits this problem the best or if I should consider using NLP.

Does anyone have a suggestion of what algorithm to use or where I can find the necessary literature to solve my problem?

Thanks for any contribution!


texts:
I have ~4000 3D images with 0,1,2 labels. 70% of the labels are zeros.
The dimension of each image is [40, 30, 30]
The values usually range between -50 to 50 and about 10% of each image has non-zero values(bigger than 10^-4).

The below is my 3D CNN model with 3 pooling layers and 3 fully connected layers.

self.conv1 = conv3d(self.X, 10, 1, 32, name='conv1')
self.norm1 = tf.contrib.layers.batch_norm(self.conv1, center=True, scale=True, scope='bn1')
self.pool1 = max_pool(self.norm1, 3, 2, name='pool1')
self.conv2 = conv3d(self.pool1,8,1,64, name='conv2')
self.norm2 = tf.contrib.layers.batch_norm(self.conv2, center=True, scale=True, scope='bn2')
self.pool2 = max_pool(self.norm2, 3, 2, name='pool2')
self.conv3 = conv3d(self.pool2,8,1,64, name='conv3')
self.pool3 = max_pool(self.conv3, 3, 2, name='pool3')
proddim = np.prod(self.pool3.get_shape()[1:], dtype=np.uint)

self.flat = tf.reshape(self.pool3, [-1,proddim ])

self.fc4 = fc(self.flat, num_output=1024, name='fc4')
self.dropout4 = tf.layers.dropout(inputs=self.fc4, rate=0.2, training=self.dropout_train, name='dropout4')
self.fc5 = fc(self.dropout4, num_output=384, name='fc5')
self.dropout5 = tf.layers.dropout(inputs=self.fc5, rate=0.2, training=self.dropout_train, name='dropout5')
self.fc6 = fc(self.dropout5, num_output=3, name='fc6')

I double, triple checked that the inputs are correct but my model always predict everything as zero.

Is it because my tensor or labels mostly have zero values? Or do I have too little data?


texts: Assessing Group Similarities and Dissimilarities Post PCA

texts:
The goal is to assess similarity and dissimilarity between 6 known groups.

The original data began with the 6 known groups and 2,700+ variables all on a scale of 0 to 100.

I have performed PCA to reduce the 2700+ variables into 5 principal components using the dudi.pca function from the ade4 package in R. Here are the Eigenvalues for the components:

      eigenvalue variance.percent cumulative.variance.percent
Dim.1   998.3274        36.635867                    36.63587
Dim.2   670.1278        24.591848                    61.22771
Dim.3   482.2372        17.696776                    78.92449
Dim.4   352.2806        12.927728                    91.85222
Dim.5   222.0270         8.147781                   100.00000

I would now like to assess the distances between the 6 known groups. Is this done as simply as generating a distance matrix using each group's coordinates for each of the principal components? If so, I am leaning towards using Manhattan distance to get the absolute distance.

Here are the coordinates of each group:

           Dim.1       Dim.2       Dim.3        Dim.4        Dim.5
Group 1    69.019038    7.940190    0.4985599  - 6.847178     0.3964117
Group 2   -16.302322  -25.965373  -29.3084201  -23.013430     9.9183010
Group 3   -26.313850   50.159662    6.9486408  -10.713924     5.2883152
Group 4   -12.800767  -26.211432   39.5067264  - 8.775551   - 8.8840592
Group 5   - 9.228404    2.648632  -20.4297314   16.685426   -26.8559444
Group 6   - 4.373694   -8.571679    2.7842244   32.664657    20.1369757

If not, what would be the appropriate way to assess individual similarity/ dissimilarity post PCA?


texts: How to implement a BiLSTM to classify speech data

texts:
I would like to know how should I provide the inputs to a BiLSTM if I am going to classify speech files (.wav) files. What is the proper way to label the data? Do I have to label each individual frame with the target label (which I think is meaningless because content of a single frame doesn't imply that the end label of the given file is what it is. Eg: First frame of utterance_0.wav doesn't directly say output label is 0)

Therefore what would be the correct way of labelling these data? And how should I maintain the .mfcc values calculated per each .wav files. (I have around 4500 files and trying to generate coefficients when we try to do the training would add extra overhead) Please correct me if I am wrong.

Since I am new to this ML approaches and training networks and RNN, I would like to have some guidance on above questions.


texts:
I have a rather large commute every day - it ranges between about an hour and about an hour and half of driving.

I have been tracking my driving times, and want to continue to do so. I am capturing the date, my time of departure, my time of arrival, the route I took (there are two or three possible ones), weather conditions (wet/dry and clear/hazy/foggy), and whether I stopped (and if so, for what reason - fuel/toilet break/food break, and for how long) for every journey to and from work.

I would like to create a system to analyse this data and suggest an optimal departure time (for the next journey) based on day of the week, weather conditions, and whether i need to stop.

Anecdotally, I can see that Tuesday mornings are worse than other mornings, the earlier I leave the more likely I am to take a toilet break or a food break, and obviously that the journey takes longer on rainy or foggy days than on clear and dry days - but I would like the system to empirically tell me that!

I assume this is a machine-learning and statistical analysis problem.

However, I have absolutely no knowledge of machine-learning, or statistical methods.

What statistical methods should I use to do this kind of analysis to the point where the data will lead to suggestions like "tomorrow is Tuesday and it is going to rain, so you must leave home between 7.50 and 8.00, and take route XYZ, to get the optimal driving time. Oh and chances are you will need a toilet break - and I have factored that in"? (assume that I manually enter tomorrow’s weather forecast - I’ll look into integrating with a weather service later)

Note that this is life-hacking for me, trying to optimise the hell out of a tedious process, and it is very personal - specific to me and my habits, specific to this route, and specific to the morning/evening commute times. Google Maps with Traffic, TomTom with IQ, and Waze do very well in the more open-ended situations of ad-hoc driving-time prediction. Even Apple is happy to tell me on my iPhone notification screen how long it will take me to get home if I leave right now.

Also note, it appears to me that traffic is not a consideration - that is to say, I do not think I need to know the actual traffic conditions - traffic is a function of day of the week and weather. For example, there are more people on the roads on Monday and Tuesday mornings, and people drive more slowly, and more people are in cars (opting to drive instead of cycle or take public transport) when it rains.

To what extent can I let the data do all the talking? I have a somewhat ambiguous hidden agenda which may not be apparent from the data;

I should be at work at 9.30 (i.,e. 9.15 +/- 15 minutes) every day, but the occasional 10am arrival is OK
I want to leave home as late as possible, and yet arrive at work as early as possible
I want to leave work as early as possible, and yet have done at least 8 hours’ work
it is OK for me to, say, leave half an hour early on one day but stay late on another to compensate

I think I can come up with a procedural formula that can encompass all of these rules, but my gut feeling is that statistical analysis can make it a lot smarter.

Apart from the methods of analysis, the technology stack is not an issue. Java is my language of choice - I am quite familiar with programming in it, and in creating web applications.

Assuming that it is possible, are there Java libraries that can provide the requisite methods?

What limitations are there? I want to keep capturing more and more data every day, making the data set bigger, hopefully, making the prediction more accurate.

What other ways are there to do it? Can I push this data into, say, Wolfram Programming Cloud, or maybe something Google provides to get the desired results?


texts: Tensorflow - Manually decay Adam optimizer

texts:
I've been experimenting with reinforcement learning and using the train_on_batch method of tf.keras.models.Model to update the models.  At this point I've done a fair amount of reading on RL but haven't seen anything on learning rate decay specific to RL.  Trying to read a little more about learning rate decay and Adam makes me think that I probably don't fully understand how various optimizers operate over batches in Tensorflow.

Taking a step back from RL, it's pretty evident that the effective learning rate decreases over the batches in each epoch with a vanilla deep learning model.  That is, towards the beginning of an epoch, the loss jumps more significantly and then stabilizes towards the end of the epoch.  This can be seen whether using Adam, SGD, etc.

Having read about learning rate decay when I first started with deep learning and after seeing this stabilizing phenomenon, I assumed that Tensorflow was decaying the learning rate automatically under the hood, however, that doesn't seem to be the case based on the answers here:

https://stackoverflow.com/questions/33919948/how-to-set-adaptive-learning-rate-for-gradientdescentoptimizer.

Looking at SGD specifically, I don't see where the default SGD settings allow for any kind of decay, but I see the same stabilization towards the end of an epoch with SGD as with Adam (Adam does have decay parameters).  Here are the SGD defaults:

tf.keras.optimizers.SGD(
    learning_rate=0.01, momentum=0.0, nesterov=False, name='SGD', **kwargs
)

So my questions are:

What's driving this stabilization towards the end of epochs in vanilla deep learning models?  What is Tensorflow doing under the hood?
How can I replicate this when using the train_on_batch method for reinforcement learning with Adam?

To clarify on #2, I think manually decaying SGD would be pretty straightforward, but because Adam has additional parameters, replicating whatever decay is happening over batches with model.fit might be more complicated, depending on the answer to #1.


texts: Performance difference between decision trees and logistic regression when one of the features is a string

texts:
I have a set of features, one of which is a string. I convert the string to an integer by treating the string as a base 36 number (I only use the first 13 characters). Then I can use DecisionTrees since in the sklearn implementation you need to convert it to a number. When I tried a different model, say Logistic Regression, performance drops drastically, say from 80% to 30% accuracy.

I might have accepted this result if I had been able to use the strings as such in the DecisionTrees model, but since I used the same string to integer conversion for both models, why such great difference?

I cannot go into the details, but let me provide you with an analogy. Let's say you are classifying millions of objects by their usefulness. So you say hammers are 4, screwdrivers 6, washers 10, etc. Of course you have more than one screwdriver, and sometime you forget and give it a value of 5, or something else. The model goes through millions of example, and then makes a prediction about the number for each object. I converted the names into integers, as I explained, and decision trees gives me an 80% accuracy, linear regression 30%. I assume that the problem is that linear regression tries to figure out some mathematical rule that does not exist. But why is decision trees immune from this problem?


texts: What is dilated pooling and how it works mathematically?

texts:
While I understand the concept of dilated convolution as there are lot of papers explaining about it, I have heard less about dilated pooling.

Can someone explain what it is?
What is the internal implementation of it? Preferably with an example.


texts: What visualization technique to best describe a recommendation dataset?

texts:
I've written a simple recommender which generates recommendations for users based on what they have clicked. The recommender generates a data file with the following format:

userid,userid,simmilarity (between 0 and 1 - closer to 0 the more similar the users)
a,b,.2
a,c,.3
a,d,.4
a,e,.1
e,b,.3
e,c,.5
e,d,.8

I've looked at some graphs, but I'm not sure which one to use, or if are there other ones that will better display the user similarities from the dataset above. Any suggestions?

I'm aiming this visualization at business users who are not at all technical. I would just like to show them an easy to understand visual that details how similar some users are and so convince the business that for these users the recommendation system is useful.

@Steve Kallestad do you mean something like this :


texts: If categorical variable has some hierarchy, should I just label them or split into dummy variables (One-Hot encode)?

texts:
I have a column which has 5 unique categories. There's a hierarchy between these categories (Best > good > OK/Not Sure > Bad > Worst)

In this case, should I label them based on hierarchy like:

Best: 4

good: 3

OK/Not Sure: 2

Bad: 1

Worst: 0

Or should I perform one-hot encoding/Dummy encoding? What's the intuition behind how the model perceives the two things?

Note: I should also mention that this is for predictive modeling using Logistic Regression.


texts: How to build classifier if only 1 image per class available

texts:
If I have only 1 image for each of 10 classes, what is the best way to build an image classifier? The images themselves are large (1200x1600) and of good quality. For example:

Or similar images from https://www.pexels.com/search/flowers/

From what I have read, neural networks need large amount of data for training. Which other machine learning methods can be used in such a situation? Basically, image similarity needs to be assessed. Can support vector machines be used here?


texts:
I have a predictive model which I trained on a training set. I have written it in R. Now I want to deploy it as a web service so anyone can just input the data into it and get the output from the predictive model.

I wanted to use Azure ML for deploying. I wanted to know whether I can drag and drop my already created/custom trained model to Azure ML studio instead of re-training it there ? I know we can train and save models in AML Studio but I am not sure about adding already created models and using them in AML solution. Help regarding this will be appreciated.


texts: Merging information of rows with the same date

texts:
My data looks like this:

date, cardio_time, muscles, muscle_time, stretch_time
2018-01-01, 0, "biceps / lats", 40, 5
2018-01-02, 30, "", 0, 10
2018-01-03, 0, "lats / calf", 41, 6
2018-01-03, 30, "hamstring", 4, 5
2018-01-04, 0, "biceps / lats", 42, 8

I would like to merge those rows with the same date, and save the info from both rows. My data would look like this after transformation, note the 3rd of January has been changed:

2018-01-01, 0, "biceps / lats", 40, 5
2018-01-02, 30, "", 0, 10
2018-01-03, 30, "lats / calf / hamstring", 45, 11
2018-01-04, 0, "biceps / lats", 42, 8

I think I could use a for loop that checks if the date on row i is the same as on row i-1, and if it is not, check the next row, but if the rows do have the same date, merge the rows together by doing something like this:

# set default value to 1 exercise per row
df['nr_excercises'] = 1
# for loop
for i in range(1, T):
    if df.index[i] == df.index[i-1]:
      # set nr of nr_excercises to 2
      df.iloc[i, nr_excercises] = 2
      # create temp variables that hold info from both rows
      cardiotimetot = df[i, cardio_time] +  df[i-1, cardio_time]
      stretchtimetot = df[i, stretch_time] +  df[i-1, stretch_time]
      etc...
      # save temp variables to i
      df.iloc[i, cardio_time] = cardiotimetot
      # drop row i-1
      df = df.drop[df.index[i-1]] # I think this is correct

Question: Is this a good approach? Is there a better way to do it?

Maybe, the code would be faster if I first use .groupby(df.index).size() to find out which days have multiple entries, and then only apply the for loop to this subset of df.


texts: Advice and Ideas appreciated - Machine Learning one man project

texts:
I have a project where I am supposed to start from scratch and learn how machine Learning works. So far everything is working out better than expected but I feel as I am offered to many ways to choose from.

My Project:
I have data with 700 rows and 108 columns as my features and I get pretty decent results when using a RandomForestClassifier.
By now I was using train_test_split to split my data but I was reading a lot of articles where it was recommended to split data into 3 Sets (train, dev, test).
Since I dont have that much Data I thought of using a Cross-Validation.

My Problem:
So I implemented it, but couldn't really find the difference between a CV and a train_test_split with shuffle.
Before doing so I thought I know the difference betweeen these to model-selection strategies but now I am a bit confused.

My Knowledge And Questions:
1. train_test_split has the problem that the sets can be unbalanced, so if I am unlucky I train my model only with positive or negative examples.
--> can't that be solved by using stratify=True?
2. train_test_split doesn't split the sets the same way all the time so the results aren't comparable
--> setting random_state=0 solves the problem?
3. When does it make sence for me to use CV?
4. How to understand the following picture and what benefit does it provide:

What would be a good way to proceed?

thanks a lot for all the help and time in advance!
Cheers!


texts: Equivalent of numeric encoding when rows can contain multiple values

texts:
If we have a column like:

    Name
0  Alice
1    Bob
2   Dave

then, after numeric encoding, it becomes:

    Name
0  0
1  1
2  2

What if, however, we have a column like this:

              Names
0        Alice, Bob
1  Alice, Bob, Dave
2              Dave

One way to encode it would be like this:

   Alice  Bob  Dave
0      1    1     0
1      1    1     1
2      0    0     1

However, this creates a lot of extra columns. Is there a way to encode such a column in a way that doesn't end up with loads of extra columns, just like using numeric encoding instead of one-hot-encoding prevents too many columns from appearing in a column like the first one I showed?

If you're using Python, here's some code to reproduce my DataFrame:

import pandas as pd
df = pd.DataFrame({'Names': ['Alice, Bob', 'Alice, Bob, Dave', 'Dave']})

Edit: the ultimate purpose of this is to then pass this through a tree-based classifier.


texts: Attention for time-series in neural networks

texts:
Neural networks in many domains (audio, video, image text/NLP) can achieve great results.
In particular in NLP using a mechanism named attention (transformer, BERT) have achieved astonishing results - without manual preprocessing of the data (text documents).
I am interested in applying neural networks to time-series.
However, in this domain, it looks like most people apply manual feature engineering by either:

transposing the matrix of events to hold columns for each time observation and a row for each thing (device, patient, ...)
manually generating sliding windows and feeding snippets to an RNN/LSMTM.

Am I overlooking something? Why can't I find people using attention? Wouldn't this be much more convenient (automated)?


texts: HIgher Order Interaction Variables. How to use them in model?

texts:
http://stats.idre.ucla.edu/stata/webbooks/logistic/chapter2/

This above website gives an example about using interaction effect between variables. Interacting variables are multiplied and used in the model.However I have read on Kaggle on using 3 or 4-way interaction effect. I use Xgbfi to find interactions but how do I use it in model ?

I have read on kaggle about hashing , bayesian coding and other few techniques. Kindly elaborate on using such 3 or 4-way interaction among Categorical , Numerical variables and between Cat - Num Vars in my model. Example codes using Python or R is also highly welcomed. Thank You.


texts: Resize instead of transposed convolutions

texts:
I'm trying to build a decoder version of ResNet, i.e. one that goes from the prelogits layer and attempts to recreate the image. I can get it working by using transposed convolutions, but the quality of reconstructed images isn't that great (checkerboard artifacts). I've read that the state of the art is to use UpSampling (i.e. resize) followed by a convolutional layer instead. Whenever I try this my network no longer converges and the images produces are all white/black.

What am I missing?


texts: What does it mean when someone says "Most of the data science algorithms are optimization problems"

texts:
I was trying to understand the Gradient Descent algorithm from this article and the author says

  Most of the data science algorithms are optimization problems

I come from software engineering background trying to get a basic understanding of data science. Can someone help me explain what this means?


texts: Trying to understand free-energy equations in a Karl Friston neuroscience article

texts:
I am trying to understand a neuroscience article:

Friston, Karl J., et al. "Action and behavior: a free-energy formulation." Biological cybernetics 102.3 (2010): 227-260. (DOI 10.1007/s00422-010-0364-z)

In this article, Friston gives three equations that are, as I understand him, equivalent or inter-convertertable and refer to both physical and Shannon entropy. They appear on page 231 of the article as equation (5):

  The resulting expression for free-energy can be expressed in three ways (with the use of the Bayes rules and simple rearrangements):

  • Energy minus entropy

  • Divergence plus surprise

  • Complexity minus accuracy

  Mathematically, these correspond to:

The things I am struggling with at this point are:

the meaning of the || in the 2nd and 3rd versions of the equations;
and the negative logs.

Any help in understanding how these equations are actually what Fristen claims them to be would be greatly appreciated. For example, in the 1st equation, in what sense is the first term energy, etc?


texts:
I'm experiencing a very strange behavior in training the following NN model for multiclass classification:

METRICS = [
    keras.metrics.AUC(name='auc')
]

model = keras.Sequential()
model.add(layers.Dense(hidden_units, activation='relu', kernel_regularizer=l2(0.1), input_shape=(input_len,)))
model.add(keras.layers.BatchNormalization())
model.add(keras.layers.Dropout(dropout_rate))

for i in range(hidden_layers-1):
    model.add(layers.Dense(hidden_units, activation='relu', kernel_regularizer=l2(0.1)))
    model.add(keras.layers.BatchNormalization())
    model.add(keras.layers.Dropout(dropout_rate))

model.add(keras.layers.Dense(output_len, activation="softmax"))

model.compile(
    optimizer=tf.keras.optimizers.Adamax(learning_rate=lr),
    loss='categorical_crossentropy',
    metrics=METRICS)

Testing different parameters combinations, my model always overfits, but the test accuracy is always lower than the train accuracy of 0.001, in terms of AUC.
The following is an example of output obtained with 3 different combinations (learning rate, epochs, batch-size, hidden layers, and hidden units per each layer).

lr: 0.001, e: 10, b: 128, l: 1, u: 200
Train : 0.992
Test  : 0.991

lr: 0.001, e: 10, b: 128, l: 2, u: 200
Train : 0.984
Test  : 0.983

lr: 0.001, e: 10, b: 500, l: 1, u: 200
Train : 0.988
Test  : 0.987

lr: 0.001, e: 10, b: 500, l: 2, u: 200
Train : 0.974
Test  : 0.973

This is how I evaluate the model:

train_auc = roc_auc_score(y_train, model.predict(X_train), average='weighted')
test_auc = roc_auc_score(y_test, model.predict(X_test), average='weighted')

Please note that I'm using average='weighted' because I'm dealing with an imbalanced dataset.

I've tried training the model with and without Dropout and regularizers, but I've obtained the same strange behavior. What am I doing wrong?


texts: Correlations - Get values in the way we want

texts:
I have :

a matrix X with N lines
a vector Y

I've computed the Euclidean distance with Y for each line of X.

What I get is a vector of distances.

What I want is a vector of scores between 0 and 1, 1 meaning "very" high correlation, 0 meaning "no" correlation.

Here what I did :

I divided the vector of distances by the max distance inside it.
I get vector D.

1 - D is the final result with values between 0 and 1.

The problem is that I get many values (75%) too close to 1.
Do you think what I did is correct ?

How would you get a better result ?
(Between 0 and 1 but not everything too close to 1)

For now, I tried to take the square of the result. (To stay between 0 and 1 but to minimize the values)

Here a picture of the distance values I want to turn in a score


texts: Neural network q learning for tic tac toe - how to use the threshold

texts:
I am currently programming a q learning neural network tha does not work. I have previously asked a question about inputs and have sorted that out. My current idea to why the program does not work is to do with the threshold value. this is a neural network - q learning specific variable.

basically the theshold is a value that is between 0 and 1, you then make a random number between 0 and 1, if this random number is larger than the threshold then you pick a completely random choice, otherwise the neural network chooses by finding the largest q value.

My question is that with this threshold value, i am currently implementing it as starting at almost 0, then increasing linearly until it reaches 1 by the time the program has reached the final  iteration. Is this correct?

The reason i suspect this is incorrect is that when plotting an error graph from training the neural network, the program doesnt not learn at all, but when the threshold reaches almost 1, it starts to learn very fast, and if you run more iterations after it reaches 1, the all the game sets in the replay memory become the same and the error is basically 0 from their on in.

Any feedback is greatly appreciated and if this question in unclear in anyway just let me know and i will try and fix it. Thank you to anyone who helps out.


texts: Are mini batches sampled randomly in Keras' Sequential.fit method()

texts: When you .fit a Keras Sequential() model, you can specify a batch_size parameter. I have noteiced it is sometimes defined independently from the actual dataset size. Does it mean that mini-batches are sampled randomly, instead of scrolling down the dataset from top to bottom, slice by slice?


texts: How to calculate which word fits the best given a context and possible words?

texts:
I have this task for research purposes and searched a while for a framework or a paper which already took care of this problem.

Unfortunately I don't find anything which helps me with my problem.

I have a sentence like

if the age of the applicant is **higher** than 18, then ...

and a list of words like

higher, bigger, greater, wider ...

which are all a

Now I want to find find out, which of the given words approximately fits the best at the predefined position in the sentence.

The best fitting word in this example would be 'greater', but for example 'higher' would be also fine.
In my specific case, I want to show an error message if someone would write 'wider', because this doesn't make sense in this semantic context.

I hope that I explained my problem good enough.


texts: How to determine precision level of outcome in ML models?

texts:
I understand there are lot of ML products that can help predict the time to event. For ex, customer will purchase in the next 30 days or Patient has chances of re-admission in next 60 days etc.
But why do you think why don't people predict to the level of date?
ex: For ex, customer will make his next purchase on Feb 2nd, 2016 or patient has chance of readmitting on Mar 23rd, 2018.
Don't know from an ordinary layman perspective, I feel dates might be more useful. Of course, all our predictions are reported based on confidence Intervals.
For ex: If we know the date when the course will end, we can plan our schedule accordingly to do other tasks/sign up for new courses etc. But if we are told that your course might end anytime in next 30/60/90 days, we are at risk of wasting few days before we enroll for another course etc. Hope my example makes sense.
Can experts here suggest me on when
a) Prediction to date level might be a good thing?
b) Prediction to date level might be a bad thing?


texts:
I am working on the next event occurrence prediction task and the data is binary time series with 1 if the event occured and 0 if not. I want to predict whether the event will occur or not on the day right after the last day of train sample.

I already tried to use LSTM network but the results are bad.

Could you please recommend which model or NN architecture would be better to use?


texts: What is the relationship between the accuracy and the loss in deep learning?

texts:
I have created three different models using deep learning for multi-class classification and each model gave me a different accuracy and loss value. The results of the testing model as the following:

First Model: Accuracy: 98.1% Loss: 0.1882
Second Model: Accuracy: 98.5% Loss: 0.0997
Third Model: Accuracy: 99.1% Loss: 0.2544

My questions are:

What is the relationship between the loss and accuracy values?
Why the loss of the third model is the higher even though the accuracy is higher?


texts: Doc2vec(gensim) - How can I infer unseen sentences’ label?

texts:
https://radimrehurek.com/gensim/models/doc2vec.html

For example, if we have trained doc2vec with

"aaaaaAAAAAaaaaaa" - "label 1"

“bbbbbbBBBBBbbbb" - "label 2"

can we infer “aaaaAAAAaaaaAA” is label 1 using Doc2vec?

I know Doc2vec can train word vectors and label vectors.
Using this vectors, can we infer unseen sentences(combination of trained words) in which label?


texts: Pick a model from multiple models using a decision tree

texts:
Let us say, I have 4 classification models on a training data set of various examples. Now, I want to choose which 1 out 4 models (or what combination of the 4 models) should I choose to get the final probability for a particular test example. So, basically I want to create a meta model to get ultimate probabilities for all the test examples. I know I can combine all 4 models using regression but I am particularly interested in using a decision tree to combine all 4 models.

So, the meta decision tree model will have 4 features/variables (4 model probabilities) as per my current thoughts. Is this a right approach? Should I include more features in it?

Please explain how do I combine 4 models using a meta decision tree model to choose the best model out of 4 for a particular test example.


texts: How to handle images of different sizes that are smaller than the input layer of a deep learning model?

texts:
I am performing human awareness detection and have trained my model using transfer learning with MobileNetV2. This model expects a tensor of dimension [Null,224,224,3].
I have applied face detection using BlazeFace which uses an input of [128,128,3] on the input video stream and cropped the detected faces in order to send the cropped faces to my custom model but I am not sure what to do as the cropped images are all of varying sizes and smaller than what my model expects.
Example of a cropped face tensor
Array [
  1,
  43,
  111,
  3,
]


texts: Cameras for automatic customer service machine

texts:
For my university project, I am planning to build an automated customer service machine. One which recognizes when someone approaches the camera according to says hello, etc.

Also, I am planning to add simple speech recognition and language processing features. So my question is.

What kind of camera would be suitable? Is there any particular model that you recommend. I was thinking of cameras used for amazon go(as an example).


texts: is learning rate schedule a hyperparameter?

texts: I believe term ‘learning rate schedule’ is a certain solution for tuning the learning rate. But at the same time, every parameter evaluating the parameter itself can be called a hyperparameter. So can the learning rate schedule be called a hyperparameter?


texts: What can functional programming be used for in data science?

texts:
In my next academic year at university, I have the option to take a course in Advanced Functional Programming. A basic description of the course is this:

"You’ll focus on a number of more advanced functional programming topics such as: programming with effects; reasoning about programs; control flow; advanced libraries; improving efficiency; type systems; and functional pearls."

Therefore I'd like to know if functional programming is useful in Data Science. If so, why it is useful, and by extension, whether this course will ultimately be useful in the pursuit of becoming a Data Scientist.


texts: How Tensorflow text prediction predicts without softmax activation

texts:
In the Colab notebook here:  RNN text generation
in def generate_text(), there is
predicted_id = tf.random.categorical(predictions, num_samples=1)[-1,0].numpy()

I looked into tf.random.categorical here:
stackoverflow answer

and sort of understand how it works.

I tried to debug/figure out what it does with print statements:

for i in range(num_generate):
      predictions = model(input_eval)
      # remove the batch dimension
      predictions = tf.squeeze(predictions, 0)

      # using a categorical distribution to predict the word returned by the model
      predictions = predictions / temperature
      predicted_id1 = tf.random.categorical(predictions, num_samples=1)
      predicted_id2 = tf.random.categorical(predictions, num_samples=1)[-1,0]
      predicted_id3 = tf.random.categorical(predictions, num_samples=1)[0,0]
      predicted_id4 = tf.random.categorical(predictions, num_samples=1)[-2,0]
      predicted_id5 = tf.random.categorical(predictions, num_samples=1)[1,0]
      predicted_id6 = tf.random.categorical(predictions, num_samples=1)[2,0]
      predicted_id7 = tf.random.categorical(predictions, num_samples=1)[3,0]
      #predicted_id8 = tf.random.categorical(predictions, num_samples=1)[4,0]

      predicted_id = tf.random.categorical(predictions, num_samples=1)[-1,0].numpy() #index 0 for

      print("predicted_id1", predicted_id1)
      print("predicted_id2", predicted_id2)
      print("predicted_id3", predicted_id3)
      print("predicted_id4", predicted_id4)
      print("predicted_id5", predicted_id5)
      print("predicted_id6", predicted_id6)
      print("predicted_id7", predicted_id7)

      print("predicted_id", predicted_id)

This was the output:

predicted_id1 tf.Tensor(
[[19]
 [33]
 [ 3]
 [35]
 [ 4]
 [64]
 [22]], shape=(7, 1), dtype=int64)
predicted_id2 tf.Tensor(35, shape=(), dtype=int64)
predicted_id3 tf.Tensor(19, shape=(), dtype=int64)
predicted_id4 tf.Tensor(3, shape=(), dtype=int64)
predicted_id5 tf.Tensor(38, shape=(), dtype=int64)
predicted_id6 tf.Tensor(26, shape=(), dtype=int64)
predicted_id7 tf.Tensor(36, shape=(), dtype=int64)
predicted_id 29

So it looks like there is some kind of distribution and that some of the indices pick from that distribution but the actual prediction in this case of 29 does not appear in the distribution so I am confused.  Are the elements in the distribution not integer ID's of characters in the text?  The one way I learned in the Udacity DLND was to assign probabilities to the predicted next character and choose the argmax so please feel free to enlighten me.


texts:
I'm working on a Twitter classification task and while analyzing the errors I found quite a few strange predictions. I'm searching for a tool (preferably open-source) similar to https://towardsdatascience.com/how-does-bert-reason-54feb363211 that is able to compute the highest positive/negative attribution given to the words (the reason why I'm not able to use the approach presented in the aforementioned article is the price). In this way, I hope that I'll be able to better understand (and possibly correct) these misclassifications. I tried looking at the attention heads but I don't feel that I'm able to fully understand and draw conclusions based on this information.
Any help would be greatly appreciated!


texts: Is there any python framework which take a dataframe and give all important relation?

texts:
Is there any python framework which takes a dataframe and gives all important relations?

For example, feature 1 and feature 2 have strong correlation when feature 3 (nominal) is equal to a special value.


texts: Parallel processing for feature selection in microarray dataset

texts:
I want to apply feature selection on a dataset with some 30-40K columns and 100 rows ( total size: 400MB-800MB ). To decrease the time consumed for calculations involved (feature-feature), I want  to divide data in some 4-5 parts and execute all parts parallely.
Since data is not huge in size, I am avoiding to use Hadoop.
What can be the option for this parallelization.. ( multi-threading , GPU or anything else) ??


texts: How to interpret shapley force plot for feature importance?

texts:
I am trying to practice and learn shapley value approach to explain my predictions on a binary classification problem. However am having difficulty in understanding the below plot.

1) Does it indicate the day_2_balance influences prediction to 1? or does blue values leads to prediction 1

2) What about the axis scale? (-4.357 to 5.643)

3) What does base value mean?

4) When I hover around the pink color , I see few more column names with some values. What do they indicate?

5) does the size of features represent their importance? Meaning PEEP_min=5 has a larger size than other features?

6) What does higher to lower and lower to higher indicate?

Can someone help me with this?


texts: How to interpret feature weight coefficients in logistic regression for text classification?

texts:
I am working on a simple text classification problem where I have as inputs tweets and as class whether that tweet contains fake news or not (0 is real news, 1 is fake news). I have trained a logistic regression model on some training data with labels and then I wanted to predict the label of new unseen data and also print the weights of each feature in order to explain the prediction. The way I feed the data to the logistic regression algorithm is by using a count vectorizer. I have done this for many tweets and I noticed something I find odd and that I can't explain.
For example given this tweet "Thanks @Lyricoldrap and @BridgetteWest we got this covid beat! soon!! Y'all are a blessing! Soup, crackers and more!  https://t.co/TKnOspiqaj", the logistic regression predicts the class 0, with probability 0.918.
However, when I print the weights of the features for this prediction I get the following:
features   weights     words
148       0.265943      soon
40        0.217010     covid
66        0.181601       got
18       -0.015743  blessing

This intuitively tells me that the word "soon" contributed the most to the prediction towards class 1 with the biggest weight, followed by "covid" and "got" and finally "blessing" contributed to the class of 0 (since it has a negative weight). Is this intuition correct? If it is not correct how is it possible I get as a prediction the class 0 with such high probability if three of my most important features are pushing the prediction in the opposite label? Or am I missing something here?


texts: How does recommendation by matrix factorization deal with new movies / users for which there are ratings?

texts:
Assume you have the ratings of $n$ users for $m$ movies in a matrix $R \in \mathbb{R}^{n \times m}$. You compute a representation

$$R = U \times \Sigma \times V$$

by initializing $u_i, v_j \forall i \in 1, \dots, n \forall j \in 1, \dots, m$ randomly and optimizing the following expression through gradient descent:

$$\min_{u_i, v_i} \sum_{p_{ij}} \left ( p_{ij} - u_i \cdot v_j \right )^2 \text{ with } u_i \in \mathbb{R}^{1 \times r}, v_j \in \mathbb{R}^{r \times 1}$$

This is how I understand how Simon Funk did it.

But how would you deal with a new user? How would you tell what that user likes?

(Or similarly, with a new movie?)


texts: Example of Reverse KL Divergence in Supervised Learning

texts:
It is of common knowledge that Supervised Learning uses forward KL divergence. However, I would like to use Reverse KL Divergence and am looking for examples of similar usage in literature. Most importantly, I would like to see how it is implemented.
If it is not feasible to implement it, I would also like to know the difficulties and approximations or workarounds to do the same.


texts: Actual problems in Data Science/Machine Learning connected with music

texts:
I'm on my $4^{th}$ year now and searching for a theme for my diploma in mathematics & computer science specialization.
Is there are interesting problems or fields for researches that i can explore? Probably i'd wanted to work in fields connected to art - music reccomendations e.g. (but we have Spotify sure).
I was suggested to take theme "Authorship attribution in classical music", but i think this theme is rather explored.


texts: Convolutional neural network with 1 channel images/1 input channel

texts:
I'm following a tutorial on tensorflow using a convolutional neural network for images, but I'm looking to do it with grayscale images. How would the code posted there be different if it was for grayscale images instead of colored images with 3 channels?
model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))


texts: What are examples for XOR, parity and multiplexer problems in decision tree learning?

texts:
In scikit-learn documentation and in decision tree learning Wikipedia article there is mention of "There are concepts that are hard to learn because decision trees do not express them easily, such as XOR, parity or multiplexer problems."

I can't recall knowing such problem types.

What are examples (and possibly related datasets) of such problems?


texts: How to filter Named Entity Recognition results

texts:
I have a pipeline built which at the end outputs a bunch (thousands to tens of thousands or more) of named entities. I'd like to do aggregates on those named entities (to see, e.g. how many times a specific named entity is mentioned in my corpus). A problem that I am arriving at; however, is that the named entities often don't match up with each other even though they are the same entity. For example, one instance of the named entity might be "Dr. John Smith" while another instance is "John Smith" or one instance might be "Google" while another might be "Google Inc.". This makes aggregating quite hard to do.

In order to deal with this issue and set "Dr. John Smith" to be the same entity as "John Smith", I was thinking of doing word matching between my named entities. I.e. I would check if named entity A has a word in common with named entity B and if they do set them to be the same entity. This approach is obviously seriously flawed. I will be equating "John Nguyen" and "John Smith" as the same entity even though they are obviously not. What's potentially even worse with this method though is I might run into similarity chains where I have "John Smith" linked with "Richard Smith" linked with "Richard Sporting Goods Inc." linked with "Google Inc." etc etc... While I may be willing to allow issues arising from former problem through, the latter problem appears to be catastrophic.

Are there any accepted techniques in the NLP community for dealing with this issue?


texts: How are the channels handled in CNN? Is it independently processed or fused?

texts:
Let's assume that we are talking about 2D convolutions applied on images.

In a grayscale image, the data is a matrix of dimensions $w \times h$, where $w$ is the width of the image and $h$ is its height. In a color image, we normally have 3 channels: red, green and blue; this way, a color image can be represented as a matrix of dimensions $w \times h \times c$, where $c$ is the number of channels, that is, 3.

A convolution layer receives the image ($w \times h \times c$) as input, and generates as output an activation map of dimensions $w' \times h' \times c'$. The number of input channels in the convolution is $c$, while the number of output channels is $c'$.

My confusion is that will CNN operate on the fused representation of the data if $c =2$ or 3 or 4 etc? Or does it operate on each channel at a time and then stacks the results? Say I have 4 channels each channel is a 2D matrix then would CNN internally form a fusion of the 4 channels and make some sort of a representation?


texts:
When plotting a Pie chart from a dictionary using Python's matplotlib, I get some classes that are too mashed together because of their small size. Is there a way to group the smallest values together and maybe plot them in a separate pie chart (or graduated bar) using python?

Here is the code I used:

import matplotlib.pyplot as plt

plt.style.use('ggplot')

dic = {'class a': 26.9,
       'Class b': 18,
       'Class c': 16.8,
       'Class d': 13,
       'Class e': 8.83,
       'Class f': 5.97,
       'Class g': 3.59,
       'Class h': 2.01,
       'Class i': 1.42,
       'Class j': 1.09,
       'Class k': 0.903,
       'Class l': 0.873,
       'Class m': 0.28,
       'Class n': 0.24,
       'Class o': 0.112}

labels = dic.keys()
sizes = dic.values()

fig, ax = plt.subplots()

ax.pie(sizes, labels=labels, autopct='%1.1f%%')
ax.axis('equal')
plt.tight_layout()
plt.show()
'''


texts: Using formulas as parameters in Power BI queries

texts:
I'm looking to create a parameter for a Power BI query that always represents the current UTC datetime.
How can I accomplish that?
This is the Advanced Editor that's given when I tried my approach:
<img src="https://i.stack.imgur.com/eASrz.png" alt="[![enter image description here][1]][1]
[1]: https://i.stack.imgur.com/MMM5B.png" />
I think this is not the way to use formulas in Power BI queries. But I did not yet find the proper way to do it. Does anyone know?


texts: Is it wrong to transform the target variable and test the model without dropping the column that was transformed? What's the disadvantage about it?

texts:
I have a linear regression model, I have transformed the target variable Item_Outlet_Sales into Item_Outlet_Sales_log on both training and testing dataset. I did not delete the Item_Outlet_Sales.
Here is the snippet:
#treat extreme values in Item_Outlet_Sales
train['Item_Outlet_Sales_log'] = np.log(train.Item_Outlet_Sales)
test['Item_Outlet_Sales_log'] = np.log(test.Item_Outlet_Sales)

sns.distplot(train.Item_Outlet_Sales_log);
sns.distplot(test.Item_Outlet_Sales_log);
#distribution

I dropped the target variable: Item_Outlet_Sales_log and assigned it to y
 #creating dummies for the training dataset
 X = train.drop('Item_Outlet_Sales_log', 1) #drop the log target column
 y = train.Item_Outlet_Sales_log

 X = pd.get_dummies(X)
 train = pd.get_dummies(train)
 test = pd.get_dummies(test)

Is it wrong to to do in terms of giving the model proper training? What is the disadvantage of it? Is it recommended to delete the old target variable or not?


texts: How does MITIE perform named entity recognition?

texts:
I'm trying to use MITIE to extract named entities from short text. I'm interested in entities such as dates, times, names, and locations. Out of the box, MITIE only recognises names, locations, and organisations. I'd like to train it to recognise dates, times and other categories as well. From looking at the structure of MITIE's directories and from the dlib website, I gather that this is done via an SVM. Is this correct?

With regards to adding new categories to the named entity recogniser, I have several questions:

Can this be done in an augmentative fashion? That is, given an existing NER system, can I just add categories examples and train it to recognise those as well? Or do I need to train models from scratch?
If I do need train models from scratch, what dataset can I use to do this?
Related to adding new examples, is there an online method that I can use, feeding the system new examples and categories as and when I need to?


texts: Why does LSTM performs better when the source target is reversed? (Seq2seq)

texts:
I'm currently reading this paper : https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf

Here, the author mentioned that LSTM learns better when the order of the source target is reversed. "Mapping (c,a'), (b,b'), (a,a') (a',b',c' is a translation of a,b,c each) is better." but I don't know why reversing the sentence makes it easy for SGD to map input n output sequence better.


texts: How can I calculate mean and variance incrementally?

texts:
Say I have a set S of values, and want to store in a database some summary information about that set, so that later when I acquire a new value v I can make a reasonable estimate of what the summary information would be about the set S ∪ {v} --- although by now I no longer have access to the original members of S. I'd like the summary information to include the mean and variance of these sets, and as minimal additional information as needed. A natural idea for the additional information would be S's cardinality. But I'm willing to save more complicated information about S if needed. My main constraint is to minimize the size of the retained information.

If I only cared about the mean of the sets, then storing the mean plus cardinality of S would obviously be enough. I could update with a new value by just taking a weighted average of the old mean (times the old cardinality) and the new value. But I'd like to be able to keep track of the variance of the sets too. A good estimate is enough; I don't need to be able to reconstruct what the exact mean and variance of S ∪ {v} would be.

I expect that even asking this displays how naive I am about statistics, but I'd appreciate any help. I don't know where to look for answers.


texts: Framework for Genetic Algorithms on python

texts:
I'm trying to use a framework implemented in python to use GA (Genetic algorithms) and other related algorithms . But I'm not sure about what framework to use, I've found two interesting options Pymoo and JmetalPy, the former it seems well documented it and the latter isn't looks like it (it's under construction), but the version in Java is very popular and well know. I need choose a framework to make a comparation with an algorithm made by me.
For this reason a reliable framework is needed and with small learning curve.
Do you have any good or  bad experience using it?,
Do you know another framework?
Pymoo:
Paper
Docs
JmetalPy: Paper
Docs


texts: Different number of features in train vs test when using Label Encoding

texts:
This is not a duplicate of Different number of features in train vs test
There are some categorical columns in my data, and the cardinality for each of them is large, so I chose to use LabelEncoding over OneHotEncoding. However, some categories in the validation set do not appear in the training set, and they cause my model to perform very poorly. I was scraping the internet but couldn't find a way.
Is there any way I can use to solve this issue?
Thank you in advance.


texts: Logistic Regression: Is it viable to use data that is outdated?

texts:
TLDR: Want to predict who makes the playoffs (1,0), but there are more playoff spots now than there were in the past, is it okay to use that past data?

I want to use binary logistic regression on MLB data to estimate each team's probability of reaching the playoffs this upcoming season.

There is data going back as far as the seasons of the 1870s. However, my issue is that the structure of the playoffs and baseball as a whole has changed often over the years. Specifically, the changes deal with the number of playoff spots, which is in part due to an increase in the number of teams. For example, up until 1969 there were 20 teams, and there was only the championship (World Series), so, technically, only 2 teams made it to the "playoffs". The number of playoff spots has increased gradually to its present state, which is 10, in 2012, and there are now 30 teams.

To me, it makes sense to only use data from 2012 (to 2019) since it reflects the state of the upcoming season. This gives me 240 observations, thus 80 positive outcomes for my playoff (dependent) variable. However, I have about 40 predictors after removing highly correlated ones, which means that I should have way more observations. Though I know that the number of predictors will likely decrease once I fit the model, I still fear my sample size may still be too low. This makes me consider going further back to the previous era beginning in 1994 when there were 8 playoff spots, simply for the sake of more observations.

My question is that would it be viable to use such data in a regression, given that it may not accurately reflect the circumstances of what I'm trying to estimate? Could I maybe even go back to 1969?

I found this article which is pretty much exactly what I'm trying to do, and he uses data back to 1969, but it just seems like an issue to me.


texts: How did anaek created chat bots using slack for HR Tools and how much time will it take as a fresher to make a similar product?


texts: Implementing neural Network using caltech course

texts:
I am trying to implement neural network using numpy using this lecture https://www.youtube.com/watch?v=Ih5Mr93E-2c&list=PLD63A284B7615313A&index=10 at timestamp 59:40.

In this lecture there is no separate bias vector used.

As you can see top neural node output is constant 1 and multiplied with weight.

I could implement forward propagation but I have problem in backward propagation while updating weights

I'm stuck near at "Update the weights" equation. When I multiply x vector with delta vector dimension doesn't match with weight matrix.

For example consider NN of 2,4,2 where 2 is input,4 is hidden and 2 is output layer. Now I want to update weight of hidden layer which is of 3x4 dimension(3 cuz of 1 extra constant node).Now when I multiply two vectors that is activation in layer 1(2x1) and transpose delta of hidden layer(4x1) I get 2x4 matrix which I can't use it to subtract with weights. Can somebody tell what I am doing wrong.

As I can't afford to go university so this place is only hope for me. Any clue would be great help for me. Thank you


texts: Does running a Decision Tree classifier several times help?

texts: To introduce, I am a novice in ML techniques. I recently had to write a scikit-learn based decision tree classifier to train on a real dataset. Someone suggested me that I must run mu model several thousand times and plot the accuracies on a graph. Here's the rub: I manually ran it around 20 -30 times and every time it gave the same accuracy (for both gini and entropy base). Is that wrong? Should it show slight variations every time?


texts: Exogenous Variables in Time Series Models

texts:
I found that in the forecast package in R that I can easily incorporate an exogenous variable Y in my ARIMA model meant to forecast X. While I have a general understanding of the kind of process I need to go through in order to determine the right ARIMA model on X, I'm not sure what I need to understand about Y in order to incorporate it as an exogenous variable.

I was wondering if anybody could help me understand the implications of having this variable in my model and whether there are any tests/checks I can perform to determine whether it's a good idea to have it in my model (other than my intuition and subject matter understanding). Any reading material on this would be helpful. Thanks!


texts:
I am really new to data science. Please don't mark me down as this website is my only hope of progress.
I have set of data I obtained from NASA website. When I saved it, it saved as "tsv' file. (Tab separated values). I want to open it on Matlab as a Matrix as I have a code to run on that matrix.
Basically I want to import that file to matlab and start running the code on it.
Can someone please help me or guide me in the right direction.

I tried various things such as import data, tdfread but so far nothing has worked for me. I was first trying to export the tsv file to MS Excel and then go from Excel to Matlab. That too I don't know how to do. I will give you the link of my data which I want to import on to Matlab.

The link for my data is the following.

Please take a look at http://vizier.u-strasbg.fr/viz-bin/VizieR?-source=J%2FApJS%2F209%2F31

If you just click on submit at the lower right corner, one will see the data.

a = importdata('J_ApJS_209_31_table3-150618.tsv') [This the command I used].
Error message on matlab is

    ??? Error using ==> importdata at 136
    Unable to open file.
    Error in ==> data at 1

Then I wrote a script.

    tdfread(J_ApJS_209_31_table3-150618,'\t')

Error message I get is
        ??? Undefined function or variable 'J_ApJS_209_31_table3'.

    Error in ==> data at 1


texts: How do i create a classifier on sensor data?

texts:
I am working on a indoor localization based on magnetometer.
I have 9 separate time-series datasets of sensor readings taken from coordinates 00, 01, 02, 10, 11, and so on until 22. Basically I am using my own coordinate system and gathered data. The coordinate system looks like this:
0,0 | 0,1 | 0,2

1,0 | 1,1 | 1,2

2,0 | 2,1 | 2,2

The dataset has columns timestamp, X, Y, Z and Magnitude. Here's what all my dataset looks like:

Edit:
I collected a sensor readings data at one place with coordinate say 0,0 and saved the file as  (coor00.csv) then at another place with coordinate say 0,1 and saved the file as  (coor01.csv) and so on.
What I want to predict is coordinates. I want to build a simple classifier that predicts the coordinate/place/area based on sensor readings.
Since I don't have any labels/tags so I thought of creating one myself by simply adding a column named label. This column will be my target variable. So in coor00.csv file I will add a column label which will have details of what coordinates the sensr readings were taken from and I will do the same for other datasets. Then I will combine 9 datasets into a dataframe and then run a classifier after splitting the dataset into train and test.
There are plenty resources out there, but I just want to know how/where to start. I want to know if this is the right way.
I plan on using RandomForest classifier but I would appreciate any suggestions on what kind of classifier algorithms should be used?


texts: Why the first prediction of neural network in PyTorch is slower than following predictions?

texts:
So I have ResNet50 trained to classify images.

For each prediction I track the time needed for it (input and model are moved to GPU):

    start = time.time()
    result = self.model.forward(transformed_image)
    end = time.time()
    print(end - start)

And always I get the following output:

1.0592937469482422
0.05996203422546387
0.06096029281616211
0.04996800422668457

So the first prediction is ~20 times longer than the following ones.

Why? And what happens behind the scenes when we launch prediction for the first time, using Torch?


texts:
This question pertains to L1 & L2 regularization parameters in Light GBM. As per official documentation:

reg_alpha (float, optional (default=0.)) – L1 regularization term on weights.

reg_lambda (float, optional (default=0.)) – L2 regularization term on weights

I have seen data scientists using both of these parameters at the same time, ideally either you use L1 or L2 not both together.

While reading about tuning LGBM parameters I cam across one such case: Kaggle official GBDT Specification and Optimization Workshop in Paris where Instructors are ML experts. And these experts have used positive values of both L1 & L2 params in LGBM model. Link below (Ctrl+F 'search_spaces' to directly reach parameter grid in this long kernel)

http://www.kaggle.com/lucamassaron/kaggle-days-paris-gbdt-workshop

I have seen same in XGBoost implementations.

My question is why use both at the same time in LGBM/XGBoost.

Thanks.


texts:
I am subsetting some data frames and am sure there is a better way.

Essentially, I have two data frames.  The first is the actual data.  The second has some meta data and importantly a flag on whether or not the row is in the subset I am interested in.  All I would like to do is pull out the subset and write a file.

For instance, here is my subset data frame:

     head(temp[,c(1,4:8)])
     ensembl_gene_id   FGF Secreted  SigP Receptors    TF
1 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE
2 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE
3 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE
4 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE
5 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE
6 ENSMUSG00000000001 FALSE    FALSE FALSE      TRUE FALSE

Here is my actual data:

Expt1_C1_EXONS_dxd_res_wDesc[1:5,1:5]
                                   groupID featureID exonBaseMean dispersion         stat
ENSMUSG00000000001:E001 ENSMUSG00000000001      E001    624.80240 0.04271781  1.255734504
ENSMUSG00000000001:E002 ENSMUSG00000000001      E002     30.92281 0.02036015  0.514038911
ENSMUSG00000000001:E003 ENSMUSG00000000001      E003     41.61413 0.01546023 10.105615831
ENSMUSG00000000001:E004 ENSMUSG00000000001      E004    137.47209 0.03975305  0.281105120
ENSMUSG00000000001:E005 ENSMUSG00000000001      E005     85.97116 0.05753662  0.005482149`

What I was doing is:

write.table(Expt1_C1_EXONS_dxd_res_wDesc[temp$FGF,],
            "Expt1_C1_EXONS_dxd_res_wDesc_FGF.tab",col.names=T,row.names=F,quote=F)

This is taking 20+ min per subset just to subset and write the data.  The data frame is 370,000 rows by 27 variables, so large but not huge.  I have about 30 of these to do.  Is there a more efficient way?  Note, the groupID does NOT equal the first column in my subset data frame.  In some instances the groupID contains a concatenated set of ensembl ids.  So I have preprocessed to get the temp data frame to have what I want in the same row order.

Thanks,
Bob


texts: One-hot encoding to embedded vector - BigGAN

texts:
I am trying to replicate a BigGAN architecture in Tensorflow but fail to understand exact nature of inputs.
BigGAN generator has 2 inputs, noise z that is a vector of [batch size, 120] elements drawn from normal distribution and vector Embedded(y) that is of size [batch size,128], where y is a vector of class labels.
By looking at tfhub code example
# Load BigGAN 128 module.
module = hub.Module('https://tfhub.dev/deepmind/biggan-128/2')

# Sample random noise (z) and ImageNet label (y) inputs.
batch_size = 8
truncation = 0.5  # scalar truncation value in [0.02, 1.0]
z = truncation * tf.random.truncated_normal([batch_size, 120])  # noise sample
y_index = tf.random.uniform([batch_size], maxval=1000, dtype=tf.int32)
y = tf.one_hot(y_index, 1000)  # one-hot ImageNet label

# Call BigGAN on a dict of the inputs to generate a batch of images with shape
# [8, 128, 128, 3] and range [-1, 1].
samples = module(dict(y=y, z=z, truncation=truncation))

my understanding is that y is a one-hot form and Embedded(y) is a mapping of these class labels.

What kind of functions is it?
How can I encode all possible classes to such embedding?
Is it equivalent to  tf.keras.layers.Embedding (see here)?version=stable ?
If yes, can it be implemented in tensorflow as explained here?


texts: R train(method="naive_bayes") and naiveBayes() very different performance

texts:
I am an R novice and having some difficulty. I was hoping R would be a good (flexible, easy) way to do machine learning of textual data.

A few years ago, I wrote a naive Bayesian classifier (from scratch) in Perl and I'm trying to replicate it in R (in order to extend it using all the cool toys R enables). My Perl script uses LOOCV and achieves an overall hit rate of 69.4%. Using equivalent pre-processing steps in R with naiveBayes() from the e1071 package with an 80-20 split I get hit rate of 60.3%. I can imagine this discrepancy is due to different crossvalidation methodologies.

Now, I am trying to find a way to use LOOCV to do an apples-to-apples comparison between my perl script and R. I'm using train() (from package "caret") with method="naive_bayes" (from a package called "naivebayes"). The code is below. No matter what I do, the results are horrible (chance-level, even worse than a no information model that exploits the baserates). Clearly either I'm doing something wrong or there is a bug; my prior p=0.999 that it's the former. I'm hoping one of you can give me a clue. My code is below.

The other aspect of performance is speed. Perl is interpreted like R and runs the LOOCV on a sample of 655 in a few seconds (so perhaps 163 N-1 analyses/second). R runs a single naive bayesian analysis very quickly, but train() takes the better part of an hour to run.

This code works well:

classifier <- naiveBayes(dtm_train,data_train$class)
pred <- predict(classifier, newdata=dtm_test)
#create a confusion table
table("predictions"=pred, "actual"=actual_classes$class)

This is an example of one way of using train that doesn't work well:

train_control <- trainControl(method="LOOCV",verboseIter = TRUE)

model5 <- train(class~., data=dtm_df,
                trControl=train_control,
                method="naive_bayes",
                verbose = TRUE
                )
print(model5)


texts: ANN structure for a multi-domain problem

texts:
I have certain problem that is composed of a steps A, B, C and D, i.e.:

y = (A º B º C º D) x

The analytical expression of these steps involves the use of Fourier Transform (FT) (actually the discrete variant, of course) to simplify the computations. One could say that some steps are expressed in simpler terms in the image domain and some in its Fourier dual.
Some steps are both non-linear and non-invertible.

I am interested in solving the inverse problem:

x = (A º B º C º D)⁻¹ y

Both x and y are actually 3D arrays.

Since I could generate an arbitrarily large number of instances of x and y, an Artificial Neural Network (ANN) seems like an excellent candidate for efficiently solving this problem.
While, in principle, a multi-layer perceptron could do the trick, it seems like this would be suffering from serious memory constrains.
I know that for similar problems Convolutional Neural Networks (CNNs) have been used.

To my understanding, the CNN-based approaches are exploiting the fact that some (hidden) convolutional transform of the data is sort-of-sparse and it will be modeled well by the hidden layers.
This works under the assumption that the convolution kernel is small enough that a traditional fully-connected layer results in a substantial amount of weights being 0 as soon as they are far from the position of the neuron being trained.
However, given the fact that some steps are simple in the Fourier domain, and hence the global Fourier transformation is used for some of the steps, I already know for sure that a local kernel is likely to fail.

Hence, I thought a multi-domain approach, where the image and its Fourier transform are given as input, could have better odds at modeling this transformation.
However, I am not clear how this could be modeled and implemented.

Could anyone point me out to a reference dealing with a similar problem?

The closest source I have found is this paper, but it looks to me that this is an even more complex approach (using wavelet transform) which is ultimately applied for classification (and not regression) and I find it quite hard to adapt this to my current problem.


texts: Passing a custom kernel with more than two arguments into `svm.SVC` in scikit-learn

texts:
I'm trying to use a custom kernel that accepts 3 arguments, with the SVM in sk-learn:

def k_gaussian(_x1, _x2, _sigma):
    normsq = np.square(np.linalg.norm(_x1-_x2))
    return np.exp(- normsq/(2 * np.square(sigma)))

According to the documentation, a custom kernel must only have two arguments, which the svm.SVC class will handle automatically with the given input data. We are told to pass the custom kernel in a form like:

clf = svm.SVC(kernel=my_kernel)

However, I'm working on an assignment which requires us to run experiments on SVM performance with varying values of _sigma.

How can I achieve that in this case? Can I pass in something like?:

clf = svm.SVC(kernel=k_gaussian(_sigma=2))

Would things like decorators help me here?


texts:
I have column with movie ids like this:

tt0984332
tt0984332
tt0847742
ttnanana1

I would like to convert them into numbers that can be inserted into neural network as features, like this:

0
0
1
2

How can I do that?


texts: Prioritized Experience Replay - which version is correct?

texts:
After reading a lot of stuff, I'm still not sure how to calculate the priorities for Prioritized Experience Replay (PER).
Example code taken from here
def compute_td_loss(batch_size, beta):
    state, action, reward, next_state, done, indices, weights = replay_buffer.sample(batch_size, beta)

    state      = Variable(torch.FloatTensor(np.float32(state)))
    next_state = Variable(torch.FloatTensor(np.float32(next_state)))
    action     = Variable(torch.LongTensor(action))
    reward     = Variable(torch.FloatTensor(reward))
    done       = Variable(torch.FloatTensor(done))
    weights    = Variable(torch.FloatTensor(weights))

    q_values      = current_model(state)
    next_q_values = target_model(next_state)

    q_value          = q_values.gather(1, action.unsqueeze(1)).squeeze(1)
    next_q_value     = next_q_values.max(1)[0]
    expected_q_value = reward + gamma * next_q_value * (1 - done)

    loss  = (q_value - expected_q_value.detach()).pow(2) * weights
    prios = loss + 1e-5
    loss  = loss.mean()

    optimizer.zero_grad()
    loss.backward()
    replay_buffer.update_priorities(indices, prios.data.cpu().numpy())
    optimizer.step()

    return loss

(V1) As you can see, the weights are used to calculate the loss and the prios. The loss is the mean squared error including the weights?
loss  = (q_value - expected_q_value.detach()).pow(2) * weights
prios = loss + 1e-5
loss  = loss.mean()

(V2) In "Hands-On Reinforcement Learning for Games: Implementing self-learning agents" on page 176 they use ONE value for ALL prios, coming from the first pow(2).mean():
loss  = (q_value - expected_q_value.detach()).pow(2).mean()
prios = loss + 1e-5
loss  = loss.mean()

(v3) Here loss is the mean squared error, prios are the raw errors:
terr  = (current_Q - target_Q.detach())
prios = terr + 1e-5
loss  = terr.pow(2).mean()

(v4) Here loss is the mean squared error, prios are the squared errors:
terr  = (current_Q - target_Q.detach())
prios = terr.pow(2) + 1e-5
loss  = terr.pow(2).mean()

(v5) Here loss is the mean squared error, prios are the absolute errors:
terr  = (current_Q - target_Q.detach())
prios = torch.abs(terr) + 1e-5
loss  = terr.pow(2).mean()

(v6) Here loss is the mean squared error, prios are the squared absolute errors:
terr  = (current_Q - target_Q.detach())
prios = torch.abs(terr).pow(2) + 1e-5
loss  = terr.pow(2).mean()

Which version is correct and why?
Thanks in advance!


texts:
Im trying to run a linear regression model with Years as the x variable and temperature on the y variable but I keep getting errors. I manage to run a regression model using the below code and also predicting for a future date 2040. But what Im struggling with is to change the data so its running from a specific date, 1950 instead of 1880 which is the original start of the data. How do you change the date range in a linear regression model?

sns.regplot('Year','No_Smoothing',data=df)

x=df.Year
y=df.No_Smoothing
np.array(x).reshape((-1,1))

from sklearn.linear_model import LinearRegression
reg =LinearRegression()
reg.fit(df[['Year']],df.No_Smoothing)
reg.predict([[2040]])


texts:
trying to merge 2 data frame size with (13647309, 48)
and I'm using 32 memory.

df_train = train.merge(train_lag,on=['ncodpers','int_date'], how='left')

after I run this it takes too much memory.
is there a way to minimize memory usage when merging?


texts: Can you use clustering to pick out signals in noisy data?

texts:
As my first project into data science, I would like to pick out the main clusters in noisy data. I think a good example would be trying to pick out certain links on a given StackExchange question that has a number of answers. The most common type of link is a link to a question on the SE network. The next common is either tag links, or links to user profiles. The remaining links might be random links included in posts, which is considered noise.  Ideally, I'm looking for a solution where I don't know how many clusters of links there will be ahead of time.

I've implemented my first attempt using scikit-learn and KMeans. However, it's not ideal because I appear to have to specify the number of clusters ahead of time, and I think the random, noisy links get grouped improperly. I also think it's more effective on a larger corpus compared to the relatively small one of URL tokens (though that's just a guess).

Is there a way to do this type of clustering, where the number of clusters is unknown or where one of the clusters is a sort of miscellaneous cluster containing objects that don't closely match the other clusters?


texts:
I am looking for packages (either in python, R, or a standalone package) to perform online learning to predict stock data.

I have found and read about Vowpal Wabbit (https://github.com/JohnLangford/vowpal_wabbit/wiki),
which seems to be quite promising but I am wondering if there are any other packages out there.

Thanks in advance.


texts: Can accuracy become worse on the training set with more epochs?

texts: I know that overfitting occurs when the accuracy on the training set improves but the accuracy on the validation set decrease. So, we must stop the training. I would like to know if this is a rule that more epochs always decrease the accuracy on the training set. In other words, is the accuracy function over epochs a non-ascending function? If not, then why?