texts: How do I find a repeating pattern of unknown length and start within a string

texts:
Background:

I am trying to do some analysis on our customer's around identifying early markers for debt delinquency.  We bill customers on a weekly basis and payment patterns will differ between accounts.  Good customers will normally pay Weekly, Fortnightly, 4 weekly or monthly on a 4-4-5 pattern.  Other customers will have an erratic payment pattern.

Even where the payment pattern is consistent, they may be overpaying or not be paying the balance in full so there may be an underlying rising or falling debt.

Work So Far:

To this end I have created two strings to represent:

Whether the weekly balance has decreased (v), increased (^) or not moved (-) (infer a payment has been made if balance is down or unchanged) and ...
The level of indebtedness on the account based on the weekly charge ( - for in credit, . for balance cleared, 0-9 for balance less than 10 weeks, X for balance over 10 weeks)

So I now have a number of strings that look like this (good accounts)

^v^v^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^^    231212-012-012-012-012-0120112-012-0-0120012-012-012-0120112-012-012-012-012-0123
^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^v^^^^    12-012-012-012-012-012-012-012-012-0-012-012-012-012-012-012-012-012-012-012-0123

and like this for bad/erratic accounts

^v^^^vv^^v^^^v^^^v^v^v^^^v^^^v^^^v^v^^vv^^^v^^vv^^vv^^vv^^vv^^vv^^vv^^vv^^vv^^v^    341234112312231234233412341234123423123312341233123301220122-011-011-011--00--001
v^v^v^^v^vv^vv^vv^v^^^^v^^^v^^^^v^v^^^v^^^v^^^^v^^^v^^^^v^^^v^^^v^^^^v^^^v^^^v^ XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX9XXX9XXXX9XXX9XXXX9XXX89XX89XXX89XX789X78

Next Steps:

What I want to do now is identify accounts with specific payment patterns:

Fortnightly = ^v^v
4weekly = ^^^v^^^v^^^v
Monthly = ^^^v^^^v^^^^v (note 4-4-5)

I also want to identify where those patterns have changed. Bearing in mind that on a 4 weekly or monthly pattern we may not be at the start of a pattern I also need to try and find the 'seed' week for each pattern

I will regard a pattern as consistent if I have seen it repeat at least 3 times so 12 weeks for the fortnightly pattern, 36 weeks for the 4 weekly and 36 weeks for the Monthly [monthly could be 445, 454 or 544 but must repeat] so I then want to flag up accounts that were consistent that have become erratic or changed their pattern.  I have data going back up to 5 years.

I have the data in Microsoft SQL and currently the only analysis tools I have to hand are Microsoft SQL and Excel.  I'm not looking for code, just what the next practical steps should be in my strategy to analyse the data.

I'm thinking find the first down step in the pattern and remove this and the preceding up ticks (and associated characters from the multiplier) then run the remaining string through a number of REPLACE() functions but this doesn't feel very 'smart'

Is there a better strategy to adopt as it feels like I am MUNGing the data?


texts: Test data significantly different from training data

texts:
What are the risks if the test data is significantly different from the training data?

Is the most significant problem related to both?


texts: Performance diagnostics in mxnet gluon (e.g. plotting training vs validation loss over time)?

texts: Tensorflow has tensorboard, is there any recommended way to plot classification error/loss over time in mxnet?


texts: How to use Keras Linear Regression for Multiple input-output?

texts:
I was trying to use this code. I put part of the parameter list, but as you see the error indicates that it's taking the first member of each list to put in the first row and second ones for the second row and so on ..

from pylab import *
from keras.models import Sequential
from keras.layers import Dense
from keras import optimizers

#Generate dummy data
data = [[45,45,200,300],[44.7,45.6,50,60],[9.9,10,11,12]]
y = data*5

#Define the model
def baseline_model():
   model = Sequential()
   model.add(Dense(1, activation = 'linear', input_dim = 1))
   sgd = optimizers.SGD(lr=0.2)
   model.compile(optimizer = sgd, loss = 'mean_squared_error', metrics = ['accuracy'])
   return model

#Use the model
regr = baseline_model()
regr.fit(data,y,epochs = 4,batch_size = 1)
plot(data, regr.predict(data), 'b', data,y, 'k.')

But was unsuccessful cause I get this error that less argoments were expected :

  ValueError: Error when checking model input: the list of Numpy arrays
  that you are passing to your model is not the size the model expected.
  Expected to see 1 array(s), but instead got the following list of 28
  arrays: [array([[  45.  ],
         [  44.7 ],
         [   9.9 ],
         [  65.5 ],
         [ 110.  ],
         [   2.  ],
         [   0.25],
         [  13.  ],
         [  17.  ]]), array([[  45.  ],
         [  45.6 ],..


texts:
I have transactional-level users data that includes their behaviour, such as reading articles, searching for content, posting, etc. I would like to cluster (most probably K-Means) these users based on aggregated metrics, For eg. SUM(articles read), SUM(searches), etc.
As users sign up at different times across years, they may be at very different cycles of their activity levels (first 30 days? inactive? etc.) So I was wondering what would be an appropriate time window to aggregate relative to each user. Should I just take all user activities in the past 30 days? Or for each user's first 30 days / their peak month? Or just simple averaging per month?
Averaging seems straightforward but for users who are on the site longer the average may be much lower than say, users who has been using the platform for only a month.


texts: Training error is oscillating, cross validation error is continuously decreasing

texts:
I created a two layered fully connected neural network as a part of a recommendation engine (after I use embedding layers for products and users). I have been trying to tune the hyper-parameters for the past couple of weeks now. Additionally I am using 5 fold cross validation and 'mse' as the error metric. Below is the curve for training and cross-validation error that I obtain post training on a GPU for 10 epochs.

I am not sure what really to make of this, fundamentally. The model doesn't overfit or underfit the data. What is even more perplexing is that the validation error is across five folds. Is this an inherent bias in the data and alludes to the fact that the data is structured such that a weak model does well on unseen data ?

Any pointers or comments will be greatly helpful. Thanks !


texts: Using data agumentation for a frozen pre-trained model

texts:
I was following the following article with regards to doing transfer learning:

https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html

In the section, Using the bottleneck features of a pre-trained network: 90% accuracy in a minute, the authors mentioned that: "Note that this prevents us from using data augmentation"

I am not very clear about this; is there a rule that discourages the use of data augmentation when the pre-trained model is totally frozen?


texts: find the values of theta in a cost function from andrew ng course

texts:
I was following Andrew NG ML course on coursera. I was stuck at cost function.

How can I find values for Theta-0 and Theta-1 ?

So I came to know Theta-0 as y intercept so the values must be Theta-0 = 0.5 but how can I find the value for Theta-1?

Any help is appreciated.


texts:
I am new in this community and hope to find some hints. At the moment I am a little bit confused by orange. In the documentation it is shown, that you can sort within the heat map rows or columns, but this option is not there...I only can merge by clusters.

Maybe a little bit more background. I measured different objects in one (microscopy) image and measured the mean intensity over all channels. Now I would like to find interesting cells by clustering or heat map visualisation. Due to the fact, that I have thousands of objects some sorting would maybe help to extract the information...tracing back to the image finding the cells.

All the best

Ralf


texts: What ML algo to choose to recreate x-ray human posture

texts:
I have an idea and not sure how to start. I have a x-ray image Of human skull. What kind of ML algorithm whold be the best to recreate the posture of the original skull.
How I see the algo steps.

Have an image of skull
Have an 3d model of skull
Arrange the 3d model in space and take an virtual x-ray of it.
Compare the real and virtual x-rays and make the adjustments to parameters.

I do not fully understand what ML algo can give adjust the virtual's skull posture based on the error of matched x-rays.
Thanks!


texts: Keras' Evaluate function training model on test set

texts:
I've been using Keras to do some timeseries predictive neural nets.
One thing that's had me staggered is how I was getting an almost 99% accuracy rate on very very noisy and uncorrelated data (I was looking for accuracy of around 50% to be good!)...

What I've now found, is that after I trained my model on the training set, I was casually running model.evaluate(x_test, y_test) before then running a manual backtest on the data. The manual backtest on x_test and comparing it to the error in y_test then showed almost perfect accuracy...

Am I then correct in assuming that Keras when using the evaluate() function in Keras the original model is actually updated with the testing data as well? This seems a bit strange and counterintuitive to me?


texts: Python experiment results directory structure creation/manipulation package

texts:
so we do numerous types of ML experiments using a number of frameworks.

We've ended up writing an awful lot of boilerplate :

read some configuration as to the experiment's parameters, variables, mode and so on
create all the necessary output directories that we'll need for all the different combinations of epoch, parameter, variable etc.

And then we traverse those output directories again to collate and summarise the results

1. and 2. are fragile (i.e. they break when new requirements come in e.g. a new "mode" for the experiment, that alters the directory layout), tedious to write and maintain and, well, boilerplate.

Questions:

is there some pypi or conda package that can do some of this dredgery for us?
is there a neat design pattern or idiom someone could suggest? E.g. each class that writes to disk knows where it should write to and just creates the directory as needed (like lazy evaluation or something).
does anyone instead use an object store e.g. MongoDB instead of saving everything to a multitude of directories? How has that been? I can see that having potential.

Many thanks.


texts: Creating a generic mathematical formula using a genetic algorithm

texts:
Assuming all of the following;

I have 4 known numbers, all within a 0-400 range, like this:

Variable1   Variable2   Variable3   Variable4
0-400       0-400       0-400       0-400

I know that there is a mathematical relationship between the numbers.
I would like to use a genetic algorithm (computer code) to estimate/approximiate Variable2 and Variable3 based on Variable1 and Variable4.
Also, importantly, assume that there are many input samples and that each sample will differ slightly. Thus, a genetic algorithm optimization of "a mathematical formula/algorithm" to estimate/approximate Variable2 and Variable3 in all cases becomes possible.
(In other words, the genetic algorithm will be able optimize the
mathematical formula towards the already-known Variable2 and
Variable3 across many input samples, each with a similar though
slightly different mathematical formula.)

How can I then write the following into a genetic algorithm:

Variable2=?
Variable3=?

Where ? could be any mathematical function (+/-/*/:/√/^2/cos/sin/tan/etc.) involving Variable2 and Variable3

In other words; I would like the genetic algorithm to build a generic mathematical formula.

How can I define Variable2 and Variable3 as the outcome of a mathematical formula so that estimation by a computer algorithm becomes possible?

I am not sure how to approach this. The genetic algorithm software I use can use as many variables as is needed, and they can be in disparate ranges.

So for example, I could write my algorithm like this easily;

Variable2=Variable1(op)Variable4
Variable3=Variable1(op)Variable4

Where Variable1 is the first variable for the genetic algorithm, with a range of 0-400, and Variable4 is the second variable for the genetic algorithm, with a range of 0-400, and finally (op) is the third variable for the genetic algorithm, for example with a range of 1-4 where 1 stands for +, 2 for -, 3 for *, 4 for : etc.

However, the complexity of this algorithm is very limited and crude; it is not optimized towards a nice and complex real estimation algorithm. Also, as soon as a secondary operator is introduced, for example:

Variable2=[Variable1 or Variable4](op)[Variable1 or Variable4](op)[Variable1 or Variable4]
Variable3=[Variable1 or Variable4](op)[Variable1 or Variable4](op)[Variable1 or Variable4]

The coding complexity for this would start to increase quickly, and there may be a need to use ( and ) to prioritize mathematical calculations, etc. The coding complexity for even more complex calculations becomes seemingly unmanageable.

Is there a better and more straightforward way to let the genetic algorithm approximate/estimate Variable2 and Variable3 based on Variable1 and Variable4 into an overall optimized generic mathematical formula/algorithm?


texts:
I have a data set with huge number of features ( Approximately 3000)  and a binary target variable . The reason I have too many features is because of one hot encoding  many categorical variables in my data set .
I think logistic regression might only work with small number of features .
So , given that I have many features , which algorithm should I use for better classification score ?
My aim is to increase the ROC-AUC metric for this classification task .
Is it better to use SVM or Neural networks ?


texts: How to train tensorflow's transformer model on my own data?

texts:
https://github.com/tensorflow/models/blob/master/official/transformer has an implementation of transformer model.
I want to train the model on my own data(consisting of two files, src.txt, and tgt.txt), However, I'm unable to figure out how to provide this input to the model.

I've gone through tensor2tensor and their topic on "train on your own data". I wish to tweak the architecture (not just hyperparameters) and would like to use the model present in tensorflow/models repository.
Any idea on how to go about this?


texts: How do CNNs use a model and find the object(s) desired?

texts:
Background: I'm studying CNN's outside of my undergraduate CS course on ML. I have a few questions related to CNNs.

1) When training a CNN, we desire tightly bounded/cropped images of the desired classes, correct? I.e. if we were trying to recognize dogs, we would use thousands of images of tightly cropped dogs. We would also feed images of non-dogs, correct? These images are scaled to a specific size, i.e. 255x255.

2) Let's say training is complete. Our model's accuracy seems sufficient, with no problems. From here, let's have a large, HD image of a non-occluded dog running through a field with various obstacles. With a typical NN and some data, we just take the model, cross it with some input, and bam it's going to output some class. How will the CNN view this large image, and then 'find' the dog? Do we run some type of preprocessing on the image to partition it, and feed the partitions?


texts: create a knowledge base for audio sentiment analysis

texts:
do you guys have any idea how to create a knowledge base for  audio  classification of emotions
where i am going to use it for the annotation of the data which i am going to feed it to my deep learning model
ie: how to make a set of rules to identify emotions
for example : if amplitude of the signal >0.5 then the emotion is angry ( this is very random)


texts: what is difference between Logistic regression and SGDClassifier with log loss OR SVM and SGDClassifer with hinge loss?

texts: Can we just use SGDClassifier with log loss instead of Logistic regression, would they have similar results ?


texts: What's a good method for combining data of different deep learning models?

texts: Suppose I want to predict the probability of a person to buy something. I want to analyze the person image and I can use a convolutional neural network, but I also want to input in my predictive model how many times he bought something, where he goes more frequently and so on. What is a correct way to input in a deep learning model informations from different deep learning models?


texts: Theano error when performing Linear Regression

texts:
I'm trying to perform Linear Regression using Theano, but there is something I might be missing or doing wrong because I receive an error message, here you have a reproducible example:
import theano
import theano.tensor as T
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split

X_theano, y_theano = make_regression(n_samples = 1000, noise = 10.0)

XT_train, XT_test, yT_train, yT_test = train_test_split(X_theano, y_theano, test_size=0.33, random_state=42)

X = T.fmatrix("X")
Y = T.fvector("Y")

w_init = np.random.randn(X_theano.shape[1])*0.01
b_init = 0.0

w = theano.shared(w_init)
b = theano.shared(b_init)

# hypothesis function
hypo = T.dot(X,w) + b
# cost function
cost = T.mean((hypo - Y)**2) + 0.01 * T.sum(w**2) # Regularizacion L2
w_grad = T.grad(cost, w)
b_grad = T.grad(cost, b)

learning_rate = 0.01

train_op = theano.function([X, Y], cost,
                           updates = [(w, w - learning_rate*w_grad),
                                      (b, b - learning_rate*b_grad)])

predict_op = theano.function(inputs = [X], outputs = hypo)

for i in range(1000):
    train_op(XT_train, yT_train)

But every time I try it I receive an error message:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-105-c5e46364b2ff> in <module>
     33
     34 for i in range(1000):
---> 35     train_op(XT_train, yT_train)
     36
     37 ann_predict = predict_op(XT_test)

/opt/anaconda3/lib/python3.7/site-packages/theano/compile/function_module.py in __call__(self, *args, **kwargs)
    811                         s.storage[0] = s.type.filter(
    812                             arg, strict=s.strict,
--> 813                             allow_downcast=s.allow_downcast)
    814
    815                     except Exception as e:

/opt/anaconda3/lib/python3.7/site-packages/theano/tensor/type.py in filter(self, data, strict, allow_downcast)
    138                             '"function". Value: "%s"'
    139                             % (self, data.dtype, self.dtype, repr(data)))
--> 140                         raise TypeError(err_msg)
    141                 elif (allow_downcast is None and
    142                         type(data) is float and

TypeError: Bad input argument to theano function with name "<ipython-input-105-c5e46364b2ff>:30" at index 0 (0-based).

This is my first time using Theano and I'm also trying to get familiar with Gradient Descent when it comes to artificial neural networks, so I think I'm doing something wrong, but I cannot point out what exactly.


texts: Comparing dataframe object with string value in django

texts:
I'm implementing machine learning model and using training dataset from MySQL table and all this is built upon Django. So basically all the calculations are done by converting entire data from MySQL table to dataframe.

df = pd.read_sql("select * from naivebayes_player",connection)

However, I'm facing problem in comparing dataframe column value with a string.

So I have a column named classification in MySQL table which has 2 fixed values 'RS' or 'NRS' stored in varchar(10) format. Since I've converted an entire table into dataframe whenever I calculate the count of 'RS' values in classification column in dataframe it always returns 0. But actually, there are 63 entries of 'RS'.

total_RS = df['classification'][df['classification']=='RS'].count()

In above line of code I'm trying to find out all records where classification is 'RS' which should be 63 but I'm getting 0. What am I doing wrong?

I have tried above code when reading data from CSV instead of MySQL table and everything worked fine.


texts:
I have a dataframe in R containing  a variable for programming languages. This is extracted from a multiple choice questionnaire in a survey. As a result, any programming language may fall into the beginning, middle or end of the whole string.
I added a binary variable for any programming language. Here is a sample of my data:

language <- c(
"Ruby, Assembly, R, Go, Rust",
"Java, Javascript, Ruby, SQL",
"Java, Ruby",
"C, C++",
"PHP, Javascript, Ruby, Assembly, Swift, R, Matlab, Go, Haskell",
"R",
"Perl, Javascript, R",
"Javascript, Ruby, Bash",
"Python, PHP, Javascript",
"Java",
"Java, C"
 )

df <-as.data.frame(language,stringsAsFactors = FALSE)
df$python      <- 0
df$java        <- 0
df$c           <- 0
df$r           <- 0
df$d           <- 0

for(trow in (1:nrow(data)))
{if (length (grep(pattern="Python",x=df[trow,"language"]))  > 0)    {df[trow,"python"] <- 1}}

I have problem with one-word languages such as C, R, D and Java . I made the variables for C#, C++ , Javascript, Ruby and others using the for loop but  I have problems with these four. What regular expression can I use so that it covers Java but not Javascript (in the beginning, middle and end of the string) and covers R but not Ruby and so on? Any other solution for making the binary variables are also welcome.


texts: Hardware requirements for Linux server to run R & RStudio

texts:
I want to build a home server/workstation to run my R projects. Based on what I have gathered, it should probably be Linux based. I want to buy the hardware now, but I am confused with the many available options for processors/ram/motherboards. I want to be able to use parallel processing, at least 64GB? of memory and enough storage space (~10TB?). Software wise, Ubuntu?, R, RStudio, PostgreSQL, some NOSQL database, probably Hadoop. I do a lot of text/geospatial/network analytics that are resource intensive. Budget ~$3000US.

My Questions:
What could an ideal configuration look like? (Hardware + Software)
What type of processor?

Notes:
No, I don't want to use a cloud solution.
I know it is a vague question, but any thoughts will help, please?
If it is off-topic or too vague, I will gladly delete.

Cheers B


texts: Keras functional api explanation of activation() layer?

texts:
With the keras functional api it is possible to write something like this:

x = Activation('relu')(x)
x = Dense(8, activation='softmax')(x)

My question is whether the Activation() function is a separate Layer (equivalent to Dense(128, activation='relu') or if not... why and when this notation is used?


texts: how to evaluate top n recommendation system with movie lens dataset?

texts:
Based on my research a recommendation system are a subclass of information filtering system that seek to predict the "rating" or "preference" that a user would give to an item. And I'm currently developing a collaborative filtering recommendation system, and basically is recommends the top 'n' items to a user (I've used the user-item algorithm).

So on top of that, I will try to evaluate my recommendation system using the movie lens dataset, and basically from my research typical evaluation measures for top-N recommendation are normalized discounted cumulative gain (NDCG) and precision/recall. So my question is how can I evaluate using these metrics (or if you have any other suggestion for another metric) using the movie lens dataset (since it's a rating of users to items).

Thanks. Any suggestions are welcome.


texts: How to compare supervised learning algorithm and it's technique ensemble learning algorithm?

texts:
I have to compare Support Vector Machine and Random Forest algorithm , but i'm confused how it can be compared, like support vector machine is supervised learning algorithm and random forest is ensemble learning .
Help me out how i can compare it on which point like - in clasification , in regression .


texts: How can i set the threshold value for the target variable. For example if a target variable is chance_of_admit and it has values from 0 to 1, how can I pick a value and so that I can convert it to 0's and 1's to perform logistic regression


texts: How would I load a database with multiple inconsistent headers?

texts:
I have a database with multiple headers like so:

site_no,datetime,00060_00003
11481500,2019-10-05,7.54
[...]
site_no,datetime,00010_00001,00010_00002,00010_00003,00060_00003
11523000,2019-10-05,15.0,14.1,14.6,1920

I need to merge these into something that looks like this:

site_no,datetime,00010_00001,00010_00002,00060_00003
11481500,2019-10-05,-1,-1,7.54,-1
11523000,2019-10-05,15.0,14.1,14.6,1920

The code that I currently have to do this is:

    df = pd.read_csv(outputFileName, false_values = ["***"], header=headerlist[0] )
    i = 1
    while i+1 < len(headerlist):
        newdf = pd.read_csv(outputFileName, false_values=["***"], skiprows=headerlist[i]-1, header = headerlist[i]).head(n=(headerlist[i+1]-headerlist[i]),)
        df.join(newdf, on = "datetime", how="outer")
        i = i + 1
    df.to_csv(outputFileName)

I get an error:

KeyError: 'datetime'


texts: How to best use Large images in training set for deep learning

texts:
I would like to ask you about how I should deal with the images I have. They are really large. They have this shape: (3000, 4000, 3).

I'm working on a multilabel classification model.

And I want to know if it's wise to slice the images into equal tiles (using image_slicer) and to feed them to the model as well as resize the original images to the requisite input size of the model and feed those too.

So the training set would be: resized_original_images + tiles_of_original_image

Thank you


texts:
I have 3 process parameters that independently cycle between zero and some peak value, as shown on this filled line time-series plot:

Although not a direct control variable, the 'health' of the system can be gauged by the consistency of relative 'stagger' (or difference) between these three parameters at any single point in time. A-B, B-C, C-A.

I am struggling to find an effective way to represent these relative stagger values - on the same timeline. A simple plot of e.g. A - B cycles positive to negative but isn't very intuitive. Ideally I'd like to represent all three relative differences on one discernible plot.


texts:
I'm trying to experiment with the performance of different machine learning algorithms before and after applying feature selection.

I tested SVM, Random Forest, KNN, Linear Regression, and, Decision Tree and I used Random Forest as a feature selection method by utilizing the function (SelectFromModel) provided by sklearn.

All the algorithms improved significantly after applying feature selection, except the Decision Tree. I'm trying to find an explanation of why that happened? Does it mean that Decision Tree needs a large number of features to produce a good model?


texts: Predict whether or not a user will visit the library tomorrow using historical data


texts: After one hot encoding, instead of columns, my number of rows are increasing

texts:
This is the code:

housing_cat1 = X_train[:,[2,7,8,10,11,13,14,15,16,21,22,23,24,25,27,28,29,30,31,32,33,35,39,40,41,42,53,55,57,58,60,63,64,65,73,74,78,79]].reshape(1,-1)

from future_encoders import OneHotEncoder

cat_encoder = OneHotEncoder(sparse=False)                  # Set to True if sparse matrix is needed
housing_cat_1_1hot = cat_encoder.fit_transform(housing_cat)
housing_cat_1_1hot


texts: Behaviour of Learning Algorithms on Random Data

texts:
Suppose we collect data for 100,000 tosses of a fair coin and record "Heads" or "Tails" as the value for the attribute outcome and also record the time, temprature and other irrelevant attributes.

We know that the outcome of each toss is random so there should be no way of predicting future unlabeled data instances.

My question is how do learning algorithms (support vector machines, for example) behave when we apply them on random data such as this?


texts: In Weka, how to draw learning curve evaluated on both test and training set?

texts:
This is just for finding overfitting gap.
After initial research, I can only find method to draw learning curve using evaluation of test set.  However, I could not evaluate on training set and over the two learning curves.


texts: Which is first ? Tuning the parameters or selecting the model

texts: I've been reading about how we split our data into 3 parts; generally, we use the validation set to help us tune the parameters and the test set to have an unbiased estimate on how well does our model perform and thus we can compare models based on the result of the test set. However, i've also read that model selection shoud be  done before tuning the parameters. I'm getting confused. Which one must be done before the other ? Is the validation set used for tuning ? If true, how are we supposed to do model selection before tuning the parameters ?


texts: Can i build an image classification model where each image has multiple labels?

texts:
If I am building a model where I need to predict the vehicle, color of it, and make of it, then can I use all the labels for a single image and build my model around it.

Like for a single image of a vehicle which is a car (car1.jpg) will have labels like - Sedan(Make), Blue(Color) and Car(Type of vehicle). Can I make a single model for this or I will have to make 3 separate models for this problem.


texts:
I have sales data of hundreds of products. How they are sold over the year in terms of units sold and revenue. Sales of each product varies largely from one another and their seasonality too. Data looks like:

In data, some products are more popular than others during some point of year,some might be popular whole year and some might be popular several times within a year. Sales of products vary from year to year and having good selling windows of one year might be days or even weeks off from the other.
So for this data, how can I figure out which product will be popular in upcoming season/month?


texts:
I was following this example online for simple text classification

And when I create the classifier object like this

from sklearn.datasets import fetch_20newsgroups
twenty_train = fetch_20newsgroups(subset='train', shuffle=True)

from sklearn.feature_extraction.text import CountVectorizer
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(twenty_train.data)
X_train_counts.shape

from sklearn.feature_extraction.text import TfidfTransformer
tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
X_train_tfidf.shape

from sklearn.naive_bayes import MultinomialNB
clf = MultinomialNB().fit(X_train_tfidf, twenty_train.target)

and run predict on the test data it gives an error

import numpy as np
twenty_test = fetch_20newsgroups(subset='test', shuffle=True)
predicted = clf.predict(twenty_test.data)
np.mean(predicted == twenty_test.target)

  ValueError: Expected 2D array, got 1D array instead:

But when I do the same thing using Pipeline it works as in the example

from sklearn.pipeline import Pipeline
text_clf = Pipeline([('vect', CountVectorizer()),
('tfidf', TfidfTransformer()),
('clf', MultinomialNB())])

text_clf = text_clf.fit(twenty_train.data, twenty_train.target)
twenty_test = fetch_20newsgroups(subset='test', shuffle=True)
predicted = text_clf.predict(twenty_test.data)
np.mean(predicted == twenty_test.target)

  Out[37]: 0.7738980350504514

Why is that?


texts: How to predict probabilities from a new data set from an already built and validated model in Python?

texts:
I have built a classification model using the following steps (and in the mentioned order) in Python -

Data cleaning - Removing unwanted variables and separating Predictor variables from response variable
Label Encoding
Standardization( StandardScaler)
Train Test Split
Smote
Model Building
Model Testing using Test data

a) Now I have a new dataset, and I want to predict using the above built model. How do I do it ? Which of the above steps should I follow and which ones should I skip ?
b) Also, is the arrangement of any of the steps aforementioned very, very wrong so that it needs to be changed ?


texts: What are some good "best practices" for dealing with large datasets in pandas and numpy? Typically i read data from csv files and use numpy and pandas to prepare them for machine learning and AI models. Sometimes I run out of memory when loading large datasets into a numpy array or a pandas dataframe, wondering what are the best ways to minimize the use of memory and avoid the "out of memory" error.


texts: Unbalanced dataset on image classification, is it better to lose samples and balance it?

texts:
I am dealing with a binary image classifier. I'm using a CNN to predict if an image is positive or negative. The problem is that the positive class represents only the 2% of the total samples. In this case, I can get lots of images, so that is not a problem. My question is about what is the best option to set up the dataset.
One option is to have an unbalanced dataset, if it is extremely unbalanced like this case, the net won't learn, it'would only predict the major class. The other option is to artificially balance the data. What I've done is preparing a dataset with a 40% of the positive class and a 60% of the negative class. This dataset does not represent the real problem and is making lose a lot of samples. Is there a better option?
Thanks.


texts: Gradient descent around optimal loss surface

texts:
All the loss surface used in examples have some of bowl shape that decrease drastically far from the optimal and decrease slowly around the optimal flat point.
My questions are:

Has all the loss surface flat area around the optimal thus causing small gradient updates?

Most of loss curve when I trained some model shows some drastic decrease in loss and shows the decreasing become slow and the loss graph become flatten. Why the loss updates (amount of the gradient) become small ?


texts: Is it possible for a times series to not have any residual component after additive/multiplicative decomposition?

texts:
After conducting a times series additive decomposition using statmodels, the chart shows only a trend component, and there isn't any seasonal or residual component observed.
I was wondering how credible the results would be if I continue the analysis with such results, and also would like to know what may cause such patterns in the series. Thanks.


texts:
I trained an LSTM with a Dense layer that has ReLU as an activation function.
Training and validation loss decreases in a very good way, however predictions on the test set are bad.

My goal is to predict words order in sentences. It’s a regression task because, for N words, I want N real numbers as output that will be ranked in order to obtain integer numbers representing words position in the sentence

Example:

input: "if Google into What ? morphed GoogleOS"

output: "What if Google morphed into GoogleOS?"

Losses:

I notice that in the predictions, there are a lot of zeros.

Model architecture:

model = tf.keras.Sequential()
model.add(Masking(mask_value=0., input_shape=(timesteps, features)))
model.add(Bidirectional(LSTM(units=200, return_sequences=True,
                             kernel_regularizer=l2(0.01),
                             recurrent_regularizer=l2(0.01),
                             bias_regularizer=l2(0.01)),
                        input_shape=(timesteps, features)))
model.add(Dense(1))
model.add(Activation('linear'))

Did ReLU (negative output is transformed in zero) causes the problem?

Is it a good choice the linear activation function in this case?


texts:
I just want to get advice on how to clean GPS directory. Here is the scenario, though I can't give so much detail or data because of some rules.
I have my dataset timestamp, latitude, and longitude. When I graph it using graphing tools, there are many outliers on track. For example, there are track that has almost clean data, just small noise. But there are tracks that has duplicates, maybe, because the surveyor tends to get measurement twice or thrice.
My question is that do you know some algorithm for removing outliers and then smoothing. I had tried getting the bearing angle from one point to the next point, then filtered if bearing angle exceed some threshold. But still, there are many outliers remaining. As I observed, some points are on location which the surveyor is in resting state, but because it is not on track, it is considered outliers. Also, I have some research about Kalman filter but I can't apply it on my code. By the way, I am using Python here.
Thank you in advance for your support.


texts: What is a good reward function when objective is to minimize the average along with the variance?

texts:
I am trying to formulate a problem where we are trying to minimize the average resource allocated to different users. Due to some inherent properties of the environment, some users can be easily minimized while it is difficult for other users due to which a fairness issue arises. While the main objective is to minimize the average resource consumed by all the users, I also want to ensure that the allocation is fair so the variance of the resource allocation is less.
So is the average+variance a proper reward function? By proper I mean does it capture what I am trying to achieve - a low average while ensuring some degree of fairness? I have seen optimization problems being formulated as x*average + y*variance where x+y=1. Would this kind of formulation be better suited for my case?


texts:
Which deep learning architecture and algorithms do you most recommend for time series classification problem?
Of course LSTM, I am looking for state of the art papers.


texts:
Do you think it is possible to learn the app, how to autonomous evaluate good or bad parking of the bikes? The thing is you need to take a picture with your phone and app need to decide according to this pictures. The parking will be in the streets.
Deciding will be based on the huge database with the pictures of the bikes which parked good/bad.


texts: heterogeneous input features for estimators in sklearn.ensemble.VotingClassifier

texts:
I would like to do ensemble on my model.
Two of them are SVM and XGBoosting. SVM could not tolerate null value and XGB can do it. So I have different features for each of them. but when VotingClassifier see null (even in condition that are not input for SVM) raise error!

is there any way to ensemble in this condition?

it's not possible to impute null value


texts:
I am currently exploring anomaly detection methods for my work and, basically I have gone through Local Oulier Factor and Isolation Forests, both unsupervised methods.

Now, the thing is, there might be a chance that I do not want a point that is far away to considered as an outlier, and so I would need some sort of supervised or semi supervised method for the outlier detection.

So what I am thinking is:

1.Label a bunch of points as outlier using LOF/IF.

2.Train a classifier on top of the labels, and then make manual adjustements if needed.

Is this what is considered a semi-supervised method? Does anybody have any experience with this sort of problem that could say if I am missing something here?

Also, because I am labeling outliers the dataset will be very unbalanced. My idea is to use bagging for this. Let's say my dataset is 1% outliers, I would train 100 equally proportional models (the outliers parts remains the same on each model, but the normal points change until I go over the entirety of the dataset) and then the final prediction is a vote of all the models. Is this stupid or a good idea?


texts: is it possible to do feature selection for unsupervised machine learning problems?

texts:
I started looking for ways to do feature selection in machine learning.

By having a quick look at this  post , I made the assumption that feature selection is only manageable for supervised learning problems:

Still, I have to ask: are there methods to do feature selection without having a known variable that will be used for a classification/regression problem?


texts: Tensorflow speech-to-text training on single file with tf.nn.ctc_loss not converging

texts:
I want to train speech-to-text model on tensorflow.

Code at:
https://github.com/bikramjitroy/speech-to-text/blob/master/TrainingUnitTest-CTCTrain.py

A sample "audio.wav" and "label.txt" file also present in repo.
https://github.com/bikramjitroy/speech-to-text

I am loading a wav file. Getting 20 mfcc feature per frame. Passing input to one layer RNN (256 hidden unit) and then added a fully connected layer and then using ctc_loss.

The accuracy is not increasing even with 500 steps with one training example :

Error = 70.932137   # Expecting it should go near zero

Accuracy = 0.62068963     # expecting this will go to zero

I have tried bi-directional rnn as well but that is diverging.

What am I doing wrong in code?


texts:
I have product data and I need to classify products to categories (for example Lenovo laptop to Laptops category, etc.), each product has properties such as:

description
list with image URLs (typically 4 photos)
product-specific properties (watches have a mechanism type attribute, etc.)
manufacturer
category ID

Category ID is my target variable, do you know some resources (articles/books) where someone did something similar? I heard about the transfer learning (answer to this question Hybrid Convolutional and Conventional Neural Networks, is it a good approach?). My biggest problem is that I don't know how to connect CNN for images similarity and conventional neural network.
Thanks for the help.


texts: How to use SMOTE in Stacking in SKLearn?

texts:
I have a data set X,y and split them to train and test data.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.20, stratify = y, random_state=10). To handle imbalanced data, I wanna use SMOTE and then use classification algorithms. However, I am going to use Stacking as my classification method. I would be thankful to know when I should use SMOTE? Should I use them in defining lower-level classifiers or in higher-level classifiers?
level0 = list()
oversample = SMOTE()
RF = RandomForestClassifier(random_state=13)
pipe1 = Pipeline(steps=[('OverSampling', oversample ), ('Classifier', RF)])
level0.append(pipe1 )

DT = DecisionTreeClassifier( random_state=0)
pipe2 = Pipeline(steps=[('OverSampling', oversample ), ('Classifier', DT)])
level0.append(pipe2)

level1 = LogisticRegression
model = StackingClassifier(estimators=level0, final_estimator=level1, cv=10, passthrough = True)
model.fit(X_train, y_train)
model.predict(X_test)

Or I should use the following code?
level0 = list()
oversample = SMOTE()
RF = RandomForestClassifier(random_state=13)
level0.append(RF)

DT = DecisionTreeClassifier( random_state=0)
level0.append(DT)

level1 = LogisticRegression
model = StackingClassifier(estimators=level0, final_estimator=level1, cv=10, passthrough = True)

pipe1 = Pipeline(steps=[('OverSampling', oversample ), ('Classifier', model)])

pipe1.fit(X_train, y_train)
pipe1.predict(X_test)

Another question, we use SMOTE in the training step to have a better model. But in pipeline, the first step is using SMOTE, and I think that in prediction on test data, at first, test data is oversampled, then classification model is applied? Is it correct? I don't know how I should use SMOTE for the final prediction. I would be thankful if someone can explain it and modify my code.


texts: Problem with calculating error rate for KNN

texts:
I am trying to validate the accuracy of my KNN algorithm for the movie rating prediction.

I have $2$ vectors: $Y$ - with the real ratings, $Y'$ - with predicted ones.

When I calculate Standard Error of the Estimate (is it the one I need to calculate?) using following formula:

$$\sigma_{est} = \sqrt{\frac{\sum (Y-Y')^2}{N}}$$

I'm getting result of $\sim 1.03$. But I thought that it can't be $> 1$. If it is not, then what does this number say to me?

results = load('first_try.mat');
Y = results(:,1);
Y_predicted = results(:,2);

o = sqrt(sum((Y-Y_predicted).^2)/rows(Y))


texts:
I have a data set of 700+ mil records with a feature that should yield good predictive power. The problem is that it has far more unique values than it should. The 10k+ unique values should map to about 150. I have that list of 150 values I want them to map to. Thinking about using a distance algorithm (levenshtein?) to map unique values from data to the desired set of values. What are some other ways to think about this problem?

Ex. 'Table', 'tab', 'tbl' should all map to 'table'. I'm not about to manually build a lookup table for this process given the volume of unique values. The unique values in the data are all derived from the desired values - they are acronyms or abbreviations.


texts:
Is there a way to pass new data to fitted ARIMA model to get predictions or I need to retrain it every time?
I need to put to production and see how it trades in real life.


texts:
The following code is my implementation of neural network (1 hidden layer) trying to predict some number based on input data.

Number of input node: 11
Number of nodes in hidden layer: 11
Number of nodes in output layer: 1
m: number of training examples, here = 4527
X: [11, m] matrix
y: [1, m] matrix
w1: weights associated from input layer to hidden layer
b1: bias vector associated from input layer to hidden layer
w2: weights associated from hidden layer to output layer
b2: bias vector associated from hidden layer to output layer
alpha: learning rate
ite: number of iteration, here = 10000

Since I'm trying to predict a continuous value output, I'm using sigmoid function in input layers and identity function in output layer

def propagate(X,y,w1,b1,w2,b2,alpha,ite):
    assert(X.shape[0] == 11)
    assert(y.shape[0] == 1)
    assert(X.shape[1] == y.shape[1])
    m = X.shape[1]
    J = np.zeros(shape=(ite,1))
    iteNo = np.zeros(shape=(ite,1))
    for i in range(1,ite+1):
        z1 = np.dot(w1,X) + b1
        a1 = sigmoid(z1)
        z2 = np.dot(w2,a1) + b2

        dz2 = (z2-y)/m
        dw2 = np.dot(dz2,a1.T)
        db2 = np.sum(dz2, axis=1, keepdims=True)
        dz1 = np.dot(w2.T,dz2)*derivative_of_sigmoid(z1)
        dw1 = np.dot(dz1,X.T)
        db1 = np.sum(dz1, axis=1, keepdims=True)

        w2 = w2 - (alpha*dw2)
        b2 = b2 - (alpha*db2)
        w1 = w1 - (alpha*dw1)
        b1 = b1 - (alpha*db1)

        iteNo[i-1] = i
        J[i-1] = np.dot((z2-y),(z2-y).T)/(2*m)

    print(z2)
    return w1,b1,w2,b2,iteNo,J

I have tried both the ways (With feature normalization and scaling & without) but my cost function varies as follows with respect number of iterations (Plotted J).

On $x$-axis: Number of iteration, On $y$-axis: Error $\times 10^{12}$.

Please help!


texts:
What is the standard (or just good) open data set to train a model for question answering? Ideally, not only in English.

Thanks!


texts:
I am not sure if this is the right place to ask this question. Anyway, I am working on a Forecasting using spending data. Using autoregression, I am able to predict the following number decently well but I would like to improve this. I have two other factors that don't predict nearly as well as the lagged factor but could still be useful in the model.

My question is this: would it be valid to include those factors in the model even though I am using a lagged version of the predictor? Does an autoregression imply that the only factor to be used is the lagged version of the predicted value?

I can't show the data as it is sensitive and my question is more about how autoregression and forecasting work in general.

Thanks for any help.


texts: Why use two LSTM layers one after another?

texts:
In the example on the Keras site, seq2seq_translate.py on line 189, there is a LSTM layer after another (the first with return_sequences=True), but another example lstm_seq2seq.py which does the same thing but letter-by-letter uses only one LSTM in the encoder. My code looks like:

encoder = LSTM(latent_dim, return_sequences=True)(encoder_inputs)
encoder_outputs, state_h, state_c = LSTM(latent_dim, return_state=True)(encoder)

My question is why does the word-by-word version use two LSTM layers? And why is the return_sequences used?


texts: Would a convolution network make sense for policy based tic-tac-toe approach?

texts:
This is inspired from my previous question, comments to which made me realize that a CNN was unsuitable for the problem The CNN required over 700k training datasets while a MLP did it in in less than 50k.

Now, I'm trying to solve the next problem and need to figure out if a CNN makes sense.

CNN detail -

Input - The board as an array of 9 elements that represents the board (0=empty, 1='X', 2='O')

Output - Recommended move as a one-hot encoded array of 9 elements. The index of 1 is the recommended move (for example, in [0,0,1,0,0,0,0,0,0] the recommended move is 2)

So, basically the CNN will be trained with a dataset that consists of boards and the move that the winner of the game has made for each of the boards. Then during evaluation, it'll try to predict the best move for a given board.

Does a convolution neural network make sense for this problem?

Note: The convnet that I was going to use for this problem is the same as in my previous question


texts: Merge two datasets based on date ranges. R

texts:
My goal is to merge two datasets using date ranges. Dataset1 contains patients stays in a hospital overtime. Dataset2 contains room information overtime. My goal is to identify what type of room the stays were in my Dataset1. It can get complicated since room type in some hospitalizations can change. For example patient 101 second hospitalization was part ICU and part Emergency.
Dataset 1
  PatientID Hospital Room           StartDate             EndDate
1       101     ODCC  4SW 2020-06-04 16:21:47 2020-06-22 15:12:39
2       101     ODCC   1W 2020-06-22 15:12:40 2020-09-08 14:03:34
3       101     ODCC   1N 2020-09-08 14:03:35 2020-10-02 06:50:24
4       101     ODCC   1W 2020-10-02 06:50:25 2020-10-05 14:25:54

Dataset 2
  Hospital Room      Type    StartDT      EndDT
1     ODCC  11A     Other 2020-01-01 2021-05-12
2     ODCC   1W       ICU 2020-06-01 2020-07-30
3     ODCC   1W Emergency 2020-08-01 2021-05-12
4     ODCC   1N Emergency 2020-11-05 2021-02-07

My goal
  Patient.ID Hospital Room           StartDate             EndDate      Type    StartDT      EndDT
1        101     ODCC  4SW 2020-06-04 16:21:47 2020-06-22 15:12:39      <NA>       <NA>       <NA>
2        101     ODCC   1W 2020-06-22 15:12:40 2020-09-08 14:03:34       ICU 2020-06-01 2020-07-30
3        102     ODCC   1W 2020-06-22 15:12:40 2020-09-08 14:03:34 Emergency 2020-08-01 2021-05-12
4        101     ODCC   1N 2020-09-08 14:03:35 2020-10-02 06:50:24      <NA>       <NA>       <NA>
5        101     ODCC   1W 2020-10-02 06:50:25 2020-10-05 14:25:54 Emergency 2020-08-01 2021-05-12

Below you can find codes to replicate my datasets.
stays <- structure(list(
  PatientID = c(101, 101, 101, 101),
  Hospital = c("ODCC", "ODCC", "ODCC", "ODCC"),
  Room = c("4SW", "1W", "1N", "1W"),
  StartDate = structure(c(1591287707, 1592838760, 1599573815, 1601621425),
                        class = c("POSIXct", "POSIXt"), tzone = "UTC"),
  EndDate = structure(c(1592838759, 1599573814, 1601621424, 1601907954),
                      class = c("POSIXct", "POSIXt"), tzone = "UTC")),
  class = "data.frame", row.names = c(NA, -4L))

type <- structure(list(
  Hospital = c("ODCC", "ODCC", "ODCC", "ODCC"),
  Room = c("11A", "1W", "1W", "1N"), Type = c("Other", "ICU", "Emergency",
                                              "Emergency"),
  StartDT = structure(c(1577836800, 1590969600, 1596240000, 1604534400),
                      class = c("POSIXct", "POSIXt"), tzone = "UTC"),
  EndDT = structure(c(1620777600, 1596067200, 1620777600, 1612656000),
                    class = c("POSIXct","POSIXt"), tzone = "UTC")),
  class = "data.frame", row.names = c(NA, -4L))

Thank you!
Marvin


texts: What is the best method to *classify* series of data?

texts:
I'm working on a project for CCG (such as HearthStone, Yu-Gi-Oh!, etc.) which does classify (or give labels) deck type when user uploads their own deck.
let's assume every cards are having their own IDs and the ID would one of alphabet character (and It should be unique). now you can assume that there are 26 cards only.
and every player can make their own deck but you should keep in mind that basic form of deck structure are already given. players can just modify small amount of cards with their taste.
let me give you guys a example:
Assume that there are many basic known deck types.
every each alphabet character means card ID and order is not considered.
every deck should have 13 cards.

e.g.)

A A B B B C C C C D E E F - This deck called 'Foo'.
C C C C F F G G E E E Z Z - This deck called 'Bar'.

// and so on...

in this circumstance, I had come up that the problem is occurred here:

If user can modify small amount of cards on their deck while maintaining basic structure of their deck type, how can we make computer can classify given deck?

so I had to decide to use Similarity Measuring which just compare between given deck and pre-defined deck and calculate similarity. so we can assume which type given deck has when it have high similarity. (using Jaccard index or some else algorithm)
but the only problem is Using deck pre-defined by developer. we've assumed that there are just 26 cards above but in real case, there are like 10k~ cards and deck type are newly created in every 3~4 months. I don't even know which decks are available out there. there are literally too many deck types to pre-define on my own.  :')
and I know above method is called Supervised Learning. I think Supervised Learning is quite challenging for this case since when new deck type have created by user community, Program wouldn't classify the type of given deck correctly when given deck type is newly created recently.
that's why we should define it every single time new deck type has released. I think that's the pain in the bum. totally waste of time/effort.
so finally I've decided to use Unsupervised Learning method which don't need pre-defined data for classification. and we know that a deck is just sequence of numeric data (Card ID). all we have to do is this:
1. Somehow convert deck (or array of Card IDs) into euclidean space
2. make groups (or cluster) by using K-Means or whatever algorithm
3. Now we can just label every groups (or clusters) to define name of deck type.

This simple but this is the reason why my head is about to blow up
so my question is:

How can we convert series of Card IDs into eucliean-space of clustering? Is it even possible? If this is impossible, Is there any method to group (or make cluster) series of data?


texts: Coding Issues faced in Andrew Ng's course

texts:
I was going through Andrew's ML course on Coursera (Week 2).
I attach with this post some snaps of my codes for the optional exercise part (of the programming assignment problems) for that week, namely, the codes for featureNormalize.m (4th image from top), computeCostMulti.m (2nd image from top), and gradientDescentMulti.m (3rd image from top), along with the outputs (1st image) that I obtained on submission, and a figure depicting the convergence of gradient descent (last image). Is something wrong with the codes for featureNormalize.m and gradientDescentMulti.m? I haven't received any feedback for these two, and I don't know what's that supposed to mean. (I haven't tried Normal Equations yet, so that one's ok).

P.S, I know that it is very stupid of me to post such a specific question on this platform, but I need an answer to it, and I can't post the codes on the User's Discussion Forum for the course on Coursera, for that violates the platform's Honor-Code.


texts: What type of ANN architecture to choose?

texts: I have N number of teachers each of which has an input feature vector (25 dimensional) consisting of positive numerical values for different quality of aspects (for example, lecturing ability, knowledge capacity, communication skills, etc). I want to design a ANN to output a single quality index based on these quality features. What type of ANN architecture is appropriate for this problem?


texts:
I would like to ask you how to work on train and test dataset. I have unlabelled data. They are short text (max 100 characters) and I would need to understand their sentiment.
To do this, I am manually assigning labels (1,0,-1). However I have more than 2000 text and I would like to find a way to do it automatically, after considering a small labelled set.
What I was thinking is to split the dataset into train and test since the beginning and work with the training dataset to label data.
Unfortunately, I have not understood how to assign labels to the remaining texts, I.e. how to predict the sentiment of data in the test dataset.
Could you please tell me what would be next steps and, if you have anything that you think can be useful for a better understanding, suggesting an example to follow? Many thanks


texts: Finding roots of a non linear expression when multiplied with a linear expression

texts:
Here is a simple polynomial equation:

b^2 + 2b + 1 = 0

I could easily solve this as:

import numpy as np
from scipy.optimize import fsolve

eq = lambda b : np.power(b,2) + 2*b + 1
fsolve(eq, np.linspace(0,1,2))

Similarly I could solve any equation, that has finite number of terms. But how do I solve an equation with infinite number of terms which is given as :

The above equation could be written as :

(1 - l) * (5.5 + 4.0*l + 4*l^2 + 6*l^3 + 5*l^4 + 5*l^5 + 5*l^6 + 5*l^7 + 5*l^8 + 5*l^9 + 5*l^10   ) = 5

when n goes from 1 to 10. But I want to solve this for sufficiently large value of n such that LHS ~= RHS.

I know the values of LHS and G1 -> Ginf but cannot understand how could I compute the value of lambda here.

I tried looking at numpy polynomial functions but could not find a function that is relevant here.


texts: On coursera what exactly does Andrew Ng say in videos Lectures 60 & 61 of machine learning?

texts:
Model Selection and Train/Validation/Test Sets - Stanford University | Coursera:

At 10:59~11:10

  One final note: I should say that in the machine learning as of this practice today, there aren't many people that will do that early thing that I talked about, and said that, you know...

Is my comprehension correct?
Because English subtitles on coursera sometimes are not correct.
As I konw, here what Chinese subtitle means is opposite to what English one does.
So I am not sure whether Andrew Ng said "there aren't" or "there are"

Thanks for your reading.

I would like to ask another one.

Diagnosing Bias vs. Variance - Stanford University | Coursera:

At 02:34~02:36, what Andrew Ng said is not quite clear as well as the English subtitle.

My comprehension is as following

  If d equals 1,.... to be high training error.

It's not that complete.

Would anyone like to identify that?

Thank you...


texts: Validation Accuracy, Validation Loss and Training Loss Remain Constant

texts:
Background
Hello,
I'm new to deep learning and I recently trained a simple convolutional neural network from Francois Chollet's Deep Learning with Python book. The network was trained on 12500 images of cats and dogs. I used 6250 images in the validation set.
For my own education, I trained the network overnight for 1200 epochs and found that the training loss, validation loss and validation accuracy eventually stopped changing. See the charts below.
Main Question
My question is the following: Why does the training loss, validation loss and validation accuracy eventually stop changing?

Some Observations
Note that in the "Training and Validation Loss" chart above, both training and validation loss dips down to a minimum near the start of training before rising, then plateauing at about 200 epochs. This indicates that the cost function does not remain "stuck" at its minimum. Perhaps the cost function gets stuck at some local minima because the cost function is non-convex? Maybe I can resolve the issue by adjusting the learning rate manually?
I should also mention that I find it strange that the training loss becomes constant after about 200 epochs, even though I used data augmentation and the training accuracy never goes to a constant value.
Additional Details
The following lists some additional details about the model setup:
Batch Size: 64
Data Augmentation:
from tensorflow import keras
from tensorflow.keras import layers

data_augmentation = keras.Sequential(
    [
        layers.experimental.preprocessing.RandomFlip("horizontal"),
        layers.experimental.preprocessing.RandomRotation(0.1),
        layers.experimental.preprocessing.RandomZoom(0.2),
    ]
)

Architecture:

Model Compilation:
model.compile(loss='binary_crossentropy',
          optimizer='rmsprop',
          metrics=['accuracy'])

Please feel free to ask for more details.
Regards,
John


texts: how to run my python code on google cloud without fear of getting disconnected - an absolute beginner?

texts:
I have been trying to use python 3 for text mining on a 650 MB csv file, which my computer was not powerful enough to do. My second solution was to reach out to google cloud. I have set up my VMs and my jupyter notebook on google cloud, and it works perfectly well. The problem, however, is that I am in constant fear of getting disconnected. As a matter of fact, my connection with google server was lost a couple of time and so was my whole work.

My question: Is there a way to have the cloud run my code without fear of getting disconnected? I need to be able to have access to my csv file and also the output file.

I know there is more than one way and have read a lot of material but they are too technical for a beginner like me to understand.

UPDATE: here is how I get access to my jupyter notebook on google cloud:
1- I run my instance on google cloud
2-I click on SSH
3- in the window that appears, I type the following:

jupyter notebook --ip=0.0.0.0 --port=8888 --no-browser &

I have seen people recommend to add nohup to the beginning of the same commend. I have tried it and got this message:

nohup: ignoring input and appending output to 'nohup.out'

And nothing happens.


texts:
I'm reading Neural Networks and Deep Learning and running into trouble with the math. One of the exercises says:

Write out $a'=\sigma (wa + b)$ in component form, and verify that it gives the same result as the rule
$$\frac{1}{1 + \exp(-\sum_{j}w_jx_j - b)}$$for computing the output of a sigmoid neuron.

Not even sure where to start here. Can anyone help me out? I'd really appreciate a detailed explanation.


texts: How to estimate the variance of regressors in scikit-learn?

texts:
Every classifier in scikit-learn has a method predict_proba(x) that predicts class probabilities for x. How to do the same thing for regressors?

The only regressor for which I know how to estimate the variance of the predictions is Gaussian process regression, for which I can do the following:

y_pred, sigma = gp.predict(x, return_std=True)

In one dimension, I can even plot, how confident the Gaussian process regressor is about its prediction of different data points

How to estimate the variance of predictions for other regressors? For example, for kernel ridge regressor, multi-layer perceptron, ensemble regressors?


texts: Please help select an Algo based on Accuracy and Confusion Matrix

texts:
I am very new to Data Science would appreciate your advice big time.

Got a task: predict if a trade will be profitable or not, based on a set of data.
I have prepared, cleaned and tested data. Selected a few two-class algorithms since the answer requires a boolean result (true or false (1/0)). After testing selected two best from my opinion: Random Forest and Logistic Regression CV (cross validation). Below are the results. Can't decide which one to use in production. Also, please note: it is better not to trade than trade and loose money, therefore false negatives are better than false positives.

Random Forest results:

Logistic Regression CV (cross validation) results:

Added a sample of dataset for clarification:


texts:
I am optimizing a model using ElasticNet, but am getting some odd behavior. When I set the tolerance hyperparameter with a small value, I get
"ConvergenceWarning: Objective did not converge"

errors. So I tried a larger tolerance value, and the convergence error goes away, but now the test data consistently gives a higher root mean squared error value. This seems backwards to me, if the model does not converge, what can cause it to give a better RMSE score, or even give consistent scores? I am using this inside a gridsearch function:
No error, bad score:
GridSearchCV(cv=KFold(n_splits=4, random_state=None, shuffle=True),
             estimator=ElasticNet(),
             param_grid={'alpha': [0.01], 'fit_intercept': [True],
                         'l1_ratio': [0.75, 0.8, 0.85, 0.9],
                         'max_iter': range(350, 450, 50), 'normalize': [False],
                         'random_state': [3], 'selection': ['random'],
                         'tol': [0.1]},
             refit='nmse', return_train_score=True,
             scoring={'mae': 'neg_mean_absolute_error',
                      'nmse': 'neg_mean_squared_error',
                      'nmsle': 'neg_mean_squared_log_error', 'r2': 'r2'})

Running GridSearchCV:
C:\miniconda3\envs\tf\lib\site-packages\sklearn\linear_model\_coordinate_descent.py:529: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Duality gap: 1253.5072980364057, tolerance: 848.4484859258542
  positive)

Error, good score:
GridSearchCV(cv=KFold(n_splits=4, random_state=None, shuffle=True),
             estimator=ElasticNet(),
             param_grid={'alpha': [0.01], 'fit_intercept': [True],
                         'l1_ratio': [0.75, 0.8, 0.85, 0.9],
                         'max_iter': range(350, 450, 50), 'normalize': [False],
                         'random_state': [3], 'selection': ['random'],
                         'tol': [0.01]},
             refit='nmse', return_train_score=True,
             scoring={'mae': 'neg_mean_absolute_error',
                      'nmse': 'neg_mean_squared_error',
                      'nmsle': 'neg_mean_squared_log_error', 'r2': 'r2'})


texts:
I'm looking to replace some of my code samples which use the Census dataset. Criticism is that there is some bias and ML fairness issues around this popular dataset. What other public datasets for binary classification which include both categorical and numerical features as well as a fair amount of examples can be relevant?

Candidates are:

KDD Cup 1999


texts:
Based on the following dataframe, I am trying to create a grouping by month, type and text,
I think I am close to what I want, however I am unable to group by month the way I want, so I have to use
the column transdate. However, when I transpose this, I lose the order

df = pd.DataFrame({'date':['6/2/2017','5/23/2017','5/20/2017','6/22/2017','4/21/2017','7/2/2017','5/23/2017','5/20/2017','8/22/2017','2/21/2017'],'rev':[100,200,300,400,500,-70,-250,-200,400,500],'text':['Car','House','Car','Truck','House','Car','House','Car','Truck','House']})
df['date'] = pd.to_datetime(df['date'])
df = df.sort_values('date')
#New Column
df['transdate'] = pd.to_datetime(df['date'])
df['transdate'] = df['transdate'].dt.strftime('%B - %Y')
#second new column
df['type'] = np.where(df['rev']>0, 'positive', 'negative')

This give me this:

Then, I create a pivot table that I am transposing

df_pivot = df.pivot_table(index='transdate',columns=['type','text'],aggfunc=sum, fill_value=0).T
df_pivot

I am wondering how I can sort the first row, starting with feb 2017, then april 2017 and so on? Or, starting the other way around, aug 2017 then july 2017 ... but keeping the order of the months?

Or, will be best to do the pivot table with index date and then, do the grouping?
If this is the case, how can I do the grouping?


texts: why do we need re-scaling ? Is it necessary for all types of data including interval and frequencies

texts: Re-scaling seems to have become a fashionable technique. The methods have propped up to deal with several types of data. Moreover,it is applied without much of context. Should we use it for only interval and continuous data or score ? and what purpose does it serve exactly ?


texts: Differential Learning Rates To Train Parts of A Network Faster

texts:
So I've had a rather "out there" idea. I want to train a dense network on a regression problem based on tabular data but I'd also like it to incorporate image data. My idea was to use a CNN based model for the images and have the output of that go directly into my dense network through a single linearly activated neuron. The other inputs to my dense model will be the different columns of the table data. It's roughly based on this tutorial.
Instead of coming up with some scoring system for the images, I'd like to train the single neuron at the end of my CNN at the same time as my DNN to perform this regression. My worry is that the DNN may reduce the weight of the image input before that neuron is able to learn an effective "score" for the images. This leads me to my question, would increasing the learning rate of the single neuron or decreasing the learning rate of the rest of my DNN work to mitigate that if it occurs?
I'll add that I'm going to perform some transfer learning on the CNN before it goes into the model to help "prime" it to be able to score these pictures but once it's incorporated into the DNN I plan on only training the single neuron output of it.


texts: Training a CNN with limited weight sharing

texts:
I am currently working with speech recognition, in which i would like to try to use CNN instead of the normal feature extraction step.

I been reading this paper which proposes method using cnn. The input is a  visual representation of mel-log filter bank energies of audio files.

And the output is phoneme recognised for each third frame section a portion (a frames, b frequency_bands) of the image.

The network is a CNN, and they propose a different weight sharing - limited weight sharing, since the patterns being seeked for doesn't occur  equally anywhere on the image, but localised to certain frequency areas.

  Using separate sets of weights for different frequency bands may be more suitable since it allows for detection of distinct feature patterns in different filter bands along the frequency axis. Fig. 5 shows an example of the limited weight sharing (LWS) scheme for CNNs, where only the convolution units that are attached to the same pooling unit share the same convolution weights. These convolution units need to share their weights so that they compute comparable features, which may then be pooled together.

I am not sure I understand the concept of this weight sharing..

Should the weights be shared for each frame but limited in frequency range?

or should it both be limited for each frame and frequency range?

They made a illustration of this weight sharing:

From what i can decipher from the image - is Limited weight sharing option 2.

Each frame do not have the same weight, multiple convolution is applied on to the same frame, and the convolution for the next frame starts at a lower frequency than than the previous frame and has stride = 2. So somehow is the convolution only performed in the diagonal of the image... sounds weird?

Sounds like i've misinterpreted something incorrectly here?
Any ideas on how to implement?


texts: Comparing Dataset - Should I use the same Test dataset?

texts:
I am training ML CNN model. I want to compare different images dataset. The dataset all have different characteristics (Translated or not, Rotated or not, etc.).
I do not modify the ML model between the different dataset training.
Should I use the same Test dataset to compare them ? This dataset would not be changed through the testing and would contain data that can't be found else where. It would not be more suited for a specific training dataset.
Or should I use a Test dataset that has the same caracteristic as the Training Dataset ? So that I can compare them at their best ?
For example, if I want to compare the dataset A and B, should I use a combinaison of Test dataset A and Test dataset B ? or When testing dataset A, use Test dataset A and when testing dataset B, use Test dataset B.


texts: How are "data instances" sampled in Multiple Instance Learning?

texts:
My understanding of Multiple Instance Learning (MIL) for a weakly supervised problem, where we, instead of having a label for each data instance, we have a label for a "bag" of instances. For example in image recognition, a bag could be a full image, a single data instance is every possible region or patch in the image, and a label could be "face". This label refers to possibly one single data instance in the image (this would be the key instance, or the specific patch in the image that contains a face).
In MIL, we take instances from labeled bags, to avoid feeding the full image into a classifier. But how do we sample these instances from each bag? There are possibly thousands of instances, which is the reason in the first place we don't want to process the full image. So we select only a sub-sample for k-instances.
But how are these samples selected? This seems like a major complication, as most of the instances won't correspond to the key instances that the bag-label refers to..
Following are some publications using MIL for image classification, and the confusions arising from each one:
https://ieeexplore.ieee.org/document/7900139
--> Here they divide images into image-subsections or tiles, each one is a "data instance", and the full image from where they come from is called the bag or set. So the bag (full image) has a label, but each data instance (image subsection or tile) does not. How data instances are sampled is not clear nor explicitly written. What is also very confusing, is that in they pseudo-code, they assign labels to each data instance (x_i, y_i), which defeats the purpose of MIL, this is now ordinary supervised learning. Clearly there is some confusion in my understanding of the algorithm and idea of MIL
https://arxiv.org/pdf/1802.04712.pdf
--> Here they again specify MIL for weakly-annotated data, where only a bag is labeled, but each data-instance within the bag does not have its own individual label (contradicting the process in the former paper).
They also validate the idea of avoiding processing of a full image, by partitioning into single tiles or data-instances.
"The MIL seems to perfectly fit
medical imaging where processing a whole image consisting
of billions of pixels is computationally infeasible. Moreover,
in the medical domain it is very difficult to obtain pixel-level
annotations, that drastically reduces number of available
data. Therefore, it is tempting to divide a medical image
into smaller patches that could be further considered as a
bag with a single label"
So the alternative is using the data instances, which are cheaper to process compared to the whole image. But this must mean we use a sub-sample of data-instances, as using all data instances would be just as expensive as processing the entire image! So there must be a sampling mechanism of selecting which data instances to use. But again, no mention of it.
I am arriving at the conclusion that one of the misunderstandings I am having is the following one: In MIL, during training, we process ALL data instances within a bag/set. There is no sub-sampling involved. If a bag is an image containing thousands of tiles or data-instances, then they will all be processed. This is probably a reason why attention models are big in MIL, as they provide a way of detecting key instances in the bag


texts: Is there a relationship between LDA, linear SVMs and Perceptron?

texts:
LDA (linear discriminant analysis), SVMs with a linear kernel, and perceptrons are linear classifiers. Is there any other relationship between them, e.g.:

Every decision boundary that can be found by LDA can be found by linear SVM
Every decision boundary that can be found by linear SVM can be found by LDA.
Every decision boundary that can be found by LDA can be found by a perceptron
Every decision boundary that can be found by linear SVM can be found by a perceptron.
Every decision boundary that can be found by a perceptron can be found by LDA
Every decision boundary that can be found by a perceptron can be found by an SVM with a linear kernel

Always on the same data, of course.

For example, I think the linear SVM can find more decision boundaries than a perceptron due to slack variables. While the perceptron finds just an arbitrary hyperplane which separates the data linearly (if such a hyperplane exists), the linear SVM will always find the same hyperplane due to the optimality criterium.


texts:
I'm currently working on a project that relies on the clustering of documents into an unknown number of clusters, based on a similarity threshold (ideally using cosine distance between tf-idf vectors).
I'm attracted to elasticsearch for the project due to the 'out-of-the-box' similarity metrics provided when querying by string, however, I'm looking for some guidance as I'm very new to this. If anyone could provide any critique of the following approach, I'd be very grateful.
Is this whole approach horrible and inefficient? How feasible is this? Am I asking 'too much' of ES for this task?
Any help would be greatly appreciated, sorry for the long read. Thanks :)


texts: How is it possible to upsample 2x with a 3x3 convolution?

texts:
From the Pytorch docs on Conv2Transpose2d, the formula to compute the output of the upsampled convolution (assuming square input and no kernel dilation) is:
$$H_{out} = (H_{in} - 1) \times S - 2P_{in}+P_{out}+1.$$
Here, $H_{out}$ and $H_{in}$ are the heights of the input and output images, $S$ is stride, and $P_{in}$ and $P_{out}$ are input and output padding.
I want to 2x upsample a 256x256 image to 512x512, using a 3x3 convolution, to simulate a convolution with fractional stride of $1/2$.
Given a stride of 2, I worked out that we must have $2 p_{in} - p_{out} = 1$, so $(p_{in}, p_{out}) = (1,1)$ is a solution.
My question is: why is $p_{out}$ necessary to achieve 2x sampling here? From this PyTorch forum post, I learned that $p_{out}$ is asymmetric padding for the right and bottom sides.
However, everywhere I look online (such as slide 54 from Stanford's CS231n 2016 lecture), it says stride 2 and pad 1 produces 2x upsampling, using 3x3 convolution, with no distinction between input/output padding.
Consider the following PyTorch code:
nn.ConvTranspose2d(in_channels, out_channels, stride=2, kernel_size=3, padding=1)
Applying that function to a 256x256 image, the dimensions of the output are 513x513. It is only until I add the parameter output_padding=1 that the dimensions are properly upsampled 2x, as confirmed by the earlier equation.
I'm hoping someone can help clarify the upsampling process in better detail, because I don't understand why "stride 2, pad 1" doesn't work unless the output is padded as well.
Thanks


texts: Help with understanding cross-validation

texts:
My understanding of cross-validation is that we divide our data set into parts 1-k, then use part 1 as a validation set and parts 2-k as a training set, then use part 2 as a validation set and the remainder as a training set, etc., until we've used each part as a validation set.

What I don't totally understand is:

What is the actual goal here? I get that we take an average of the k rounds of cross-validation that we do, but what is our output? I've read that we discard the result of each round after calculating the validation error, so how do we actually produce a model from this?
What is the difference between a validation and test set, if any? i.e., we normally separate our data into training and test data, but do we then further divide our training data into validation sets, and keep our test data aside during the cross-validation process, or do we run cross-validation on the entire data set?


texts: LSTM-Model - Validation Accuracy is not changing

texts:
I am working on classification problem, My input data is labels and output expected data is labels
I have made X, Y pairs by shifting the X and Y is changed to the categorical value
Labels  Count
1      94481
0      65181
2      60448

    X   Y
    2   1.0
    1   2.0
    1   1.0
    2   1.0
    2   2.0

encoder = LabelEncoder()
test_labels = to_categorical(encoder.fit_transform(values[:,1]),num_classes=3)
train_X,test_X,train_y,test_y= train_test_split(values[:,0], test_labels,test_size = 0.30,random_state = 42)

print(train_X.shape)
print(train_y.shape)
print(test_X.shape)
print(test_y.shape)

(154076,)
(154076, 3)
(66033,)
(66033, 3)
Converting this to LSTM format
train_X = train_X.reshape(train_X.shape[0],1,1)
test_X = test_X.reshape(test_X.shape[0],1,1)

# configure network
n_batch = 1
n_epoch = 10
n_neurons = 100

ModelArchitecture
tf.keras.backend.clear_session()

model = tf.keras.models.Sequential([
    tf.keras.layers.LSTM(n_neurons, batch_input_shape=(n_batch, train_X.shape[1],train_X.shape[2]), stateful=True),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(100, activation = 'relu',kernel_regularizer=regularizers.l2(0.0001)),
    tf.keras.layers.Dense(3, activation='softmax')
])

model.summary()
model.compile(optimizer='rmsprop',
              loss='categorical_crossentropy',
              metrics=['acc'])
history = model.fit(train_X,train_y,validation_data=(test_X, test_y),epochs=n_epoch, batch_size=n_batch, verbose=1,shuffle= False)

Validation Accuracy is not Changing
Epoch 1/5
154076/154076 [==============================] - 356s 2ms/step - loss: 1.0844 - acc: 0.4269 - val_loss: 1.0814 - val_acc: 0.4310
Epoch 2/5
154076/154076 [==============================] - 354s 2ms/step - loss: 1.0853 - acc: 0.4256 - val_loss: 1.0813 - val_acc: 0.4310
Epoch 3/5
154076/154076 [==============================] - 355s 2ms/step - loss: 1.0861 - acc: 0.4246 - val_loss: 1.0814 - val_acc: 0.4310
Epoch 4/5
154076/154076 [==============================] - 356s 2ms/step - loss: 1.0874 - acc: 0.4228 - val_loss: 1.0825 - val_acc: 0.4310
Epoch 5/5
154076/154076 [==============================] - 353s 2ms/step - loss: 1.0887 - acc: 0.4208 - val_loss: 1.0828 - val_acc: 0.4310

What can be the changes to improve the model.
Update2
After Increasing the learning rate of Rmsprop to 0.5 , Below is the training loss and validation loss
Epoch 1/10
176087/176087 [==============================] - 370s 2ms/step - loss: 1.4314 - acc: 0.3594 - val_loss: 12.9380 - val_acc: 0.2739
Epoch 2/10
176087/176087 [==============================] - 369s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 4.6767 - val_acc: 0.2854
Epoch 3/10
176087/176087 [==============================] - 371s 2ms/step - loss: 1.3209 - acc: 0.3592 - val_loss: 12.9380 - val_acc: 0.2739
Epoch 4/10
176087/176087 [==============================] - 370s 2ms/step - loss: 1.3209 - acc: 0.3592 - val_loss: 4.6767 - val_acc: 0.2854
Epoch 5/10
176087/176087 [==============================] - 368s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 12.9380 - val_acc: 0.2739
Epoch 6/10
176087/176087 [==============================] - 370s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 4.6767 - val_acc: 0.2854
Epoch 7/10
176087/176087 [==============================] - 368s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 12.9380 - val_acc: 0.2739
Epoch 8/10
176087/176087 [==============================] - 372s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 4.6767 - val_acc: 0.2854
Epoch 9/10
176087/176087 [==============================] - 371s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 12.9379 - val_acc: 0.2739
Epoch 10/10
176087/176087 [==============================] - 369s 2ms/step - loss: 1.3209 - acc: 0.3593 - val_loss: 4.6767 - val_acc: 0.2854


texts: IterativeImputer - Returning -0 and other wierd results

texts:
I am using IterativeImputer to impute my dataset.
from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer

imp = IterativeImputer(random_state=0, max_iter=100, verbose=10)
imp.fit(hosp)

hosp_imputed = pd.DataFrame(imp.transform(hosp), columns=cols)

I have a boolean column "ICU" that had 8 missing values. But after imputation, it outputs very weird results. Please see the attached screenshot. Can you please let me know what am I doing wrong?


texts: How can I use a multivariable list of strings as a single X variable / feature in a Neural Network Model

texts:
Here is an example of what I am trying to do.

I am using a bunch of X variables to predict a Y.

X1 = Number of days since applying first medication
X1.1 = Total times the medication should have been applied (calculated using frequency in prescription)
X1.2 = Total times the medication was applied (calculated)

X2 = Number of days since applying first medication
X2.1 = Total times the medication should have been applied (calculated using frequency in prescription)
X2.2 = Total times the medication was applied (calculated)

X3 = Number of days since applying first medication
X3.1 = Total times the medication should have been applied (calculated using frequency in prescription)
X3.2 = Total times the medication was applied (calculated)

Y = Skin clear 0 / 1

I have some sample data as below:

Now, the X1.1 and 1.2 are descriptive of X1. As a separate input to the Neural Network, they are meaningless.

I am trying to find, how I can tell the model that X1, X1.1 and X1.2 together is a single input with multiple dimensions.

I use the below code:

classifier = Sequential()

classifier.add(Dense(units = 14, kernel_initializer = 'uniform', activation = 'relu', input_dim = 27))

classifier.add(Dense(units = 14, kernel_initializer = 'uniform', activation = 'relu'))

classifier.add(Dense(units = 1, kernel_initializer = 'uniform', activation = 'sigmoid'))

classifier.compile(optimizer = 'adam', loss = 'binary_crossentropy', metrics = ['accuracy'])

classifier.fit(X_train, y_train, batch_size = 25, epochs = 500)

Any help will be really appreciated.


texts: XGBoost hyperparameters depend on number of samples, how can I avoid constantly retraining as I collect more samples?

texts:
I'm currently using this:

XGBRegressor(
    learning_rate=0.2, n_estimators=100, max_depth=6, min_split_loss=0.1, min_child_weight=1,
    reg_alpha=2, reg_lambda=2, scale_pos_weight=1, nthread=3, subsample=0.5, colsample_bytree=0.5
)

With n samples, this works well. With 2n samples, it overfits. With n/2 samples, it underfits. I have to change the learning_rate and reg_alpha whenever I change the number of samples. Is there a systematic way to make the hyperparameters independent of the number of input samples?

I could just make the hyperparameters variables dependent on the number of samples, but I'm wondering if there's a better way to do so.


texts:
Is it possible to use a trained CNN to generate data? After training on data of X and corresponding Ys, given a new Y to generate X?
Or do I have to use GANs or RNNs? I'm still pretty new to the subject so I would be happy to have some leads, if I have to go in this direction.

Currently I am learning tensorflow  and trying to implement the "Convolutional Sequence to Sequence Learning" paper

Clarification: I want to use a CNN to generate sequences of text. I have a reallyb big database of descriptions and also one of 1000 classes, and wanted to try it out with CNNs (since im coming from image classification)


texts: Why did Tufte call this a "superbly produced duck"?

texts:
I think I understand Tufte's concept of a "Duck" -- A graphic that is taken over by decorative forms.

But I couldn't understand why he called this a duck (a "superbly produced" one at that). It seemed to me more functional than decorative.

Thoughts?


texts: PyTorch time series prediction beyond test data

texts:
I am currently playing around with pytorch models for time series prediction. I have managed to successfully run a model to predict test data. I was wondering how can I use it to predict beyond test data? I will attach my code below. I essentially want the model to continue running for say 1000 more points after the test data.

import torch
import torch.nn as nn
import torchvision.transforms as transforms
import torchvision.datasets as dsets
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import sys
from torch.autograd import Variable
from sklearn.preprocessing import MinMaxScaler

# Load the data set
data_test = np.loadtxt('data.txt')
training_set = np.reshape(data,(len(data),1))

def sliding_windows(data, seq_length):
    x = []
    y = []

    for i in range(len(data)-seq_length-1):
        _x = data[i:(i+seq_length)]
        _y = data[i+seq_length]
        x.append(_x)
        y.append(_y)

    return np.array(x),np.array(y)

sc = MinMaxScaler()
training_data = sc.fit_transform(training_set)

batch_size = 4
x, y = sliding_windows(training_data, batch_size)
train_size = int(len(y) * 0.5)
test_size = len(y) - train_size

data_X = Variable(torch.Tensor(np.array(x)))
data_Y = Variable(torch.Tensor(np.array(y)))

training_set_X = Variable(torch.Tensor(np.array(x[0:train_size])))
training_set_Y = Variable(torch.Tensor(np.array(y[0:train_size])))

testing_set_X = Variable(torch.Tensor(np.array(x[train_size:len(x)])))
testing_set_Y = Variable(torch.Tensor(np.array(y[train_size:len(y)])))

class LSTMModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):
        super(LSTMModel, self).__init__()
        # Hidden dimensions
        self.hidden_dim = hidden_dim

        # Number of hidden layers
        self.layer_dim = layer_dim

        # Inlcude dropout if you want (decided not to since it did not seem to affect results very much)
        #self.dropout = nn.Dropout(p=0.5)

        # Builds the LSTM
        self.lstm = nn.LSTM(input_dim, hidden_dim, layer_dim, batch_first=True)

        # Readout layer
        # Linear = Applies a linear transformation to the incoming data: y = xW^T + b
        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        # Initialize hidden state
        h0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_()

        # Initialize cell state
        c0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_()

        # We need to detach as we are doing truncated backpropagation through time (BPTT)
        # If we don't, we'll backprop all the way to the start even after going through another batch
        out, (hn, cn) = self.lstm(x, (h0.detach(), c0.detach()))

        # Index hidden state of last time step
        # out[:, -1, :] --> Last time step hidden state
        out = self.fc(out[:, -1, :])
        return out

input_dim = 1
hidden_dim = 50
layer_dim = 1
output_dim = 1
num_epochs = 100
lr = 0.1

model = LSTMModel(input_dim, hidden_dim, layer_dim, output_dim)

criterion =  torch.nn.MSELoss()

optimizer = torch.optim.Adam(model.parameters(), lr=lr)

decayRate = 1
lr_scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer = optimizer, gamma = decayRate)

for epoch in range(num_epochs):
    outputs = model(training_set_X)
    optimizer.zero_grad()
    loss = criterion(outputs, training_set_Y)
    loss.backward()
    optimizer.step()
    lr_scheduler.step()

model.eval()
train_predict = model(data_X)

data_predict = train_predict.data.numpy()
dataY_plot = data_Y.data.numpy()

data_predict = sc.inverse_transform(data_predict)
dataY_plot = sc.inverse_transform(dataY_plot)

dataY_plot = np.array(dataY_plot)
data_predict = np.array(data_predict)
```


texts: Knowing Feature Importance from Sparse Matrix

texts:
I was working with a dataset which had a textual column as well as numerical columns, so I used tfidf for textual column and created a sparse matrix, similarly for the numerical features I created a sparse matrix using scipy.sparse.csr_matrix and combined them with the text sparse features.

Then I'm feeding the algorithm to a gradient boosting model and doing the rest of the training and prediction. However I want to know, is there any way I can plot the feature importance, of this sparse matrix and will be able to know the important feature column names?


texts: How to properly train your Self-Organized Map?

texts:
I stumbled recently upon the Self-Organized Map, an ANN architecture used to cluster high dimensional data, while simultaneously imposing a neighbourhood structure on it. It's trained through a competitive learning approach where neurons compete to respond to a given input. The strongest responding neuron / best matching unit (BMU) is rewarded by being moved closer to the given input in the data space, as well its neighbours. However, within the literature and implementations, I find some deviations in how this training is implemented. Specifically, the influence of the BMU on its neighbors are mitigated using a neighborhood function

where d is the distance of the BMU to the input and σ(t) is a radius which is decreased during the training. Effectively, resulting in the influence of the readjustment of the BMU on its neighbourhood shrinking during training. The difference I find concerns the implementation of the shrinking of σ(t) . Most explanations and blog posts describe an exponential decrease

where λ is a decay constant which can be tuned arbitrarily. Alternatively, I find that some implementations do not really use this exponential decay, but instead use a linear interpolation of the form

where n is the number of training epochs and r is radius which is altered depending the training phase. These implementations further explicitely between a 'rough' training phase where

with e.g. SOM.dims=(100,100) being for a 100x100 sized SOM, and a 'fine-tuning' training phase where

My problem is that I do not quite understand why there seems to be this disagreement and what the 'canonical' way of training a SOM is. It certainly makes sense to divide the training into a 'rough' and a 'fine-tuning' phase, but why most newer descriptions neglect this without further discussion and only consider a single training phase with exponential decay is baffling me a bit.


texts: Is there any domain where Bayesian Networks outperform neural networks?

texts:
Neural networks get top results in Computer Vision tasks (see MNIST, ILSVRC, Kaggle Galaxy Challenge).  They seem to outperform every other approach in Computer Vision. But there are also other tasks:

Kaggle Molecular Activity Challenge
Regression: Kaggle Rain prediction, also the 2nd place
Grasp and Lift 2nd also third place - Identify hand motions from EEG recordings

I'm not too sure about ASR (automatic speech recognition) and machine translation, but I think I've also heard that (recurrent) neural networks (start to) outperform other approaches.

I am currently learning about Bayesian Networks and I wonder in which cases those models are usually applied. So my question is:

Is there any challenge / (Kaggle) competition, where the state of the art are Bayesian Networks or at least very similar models?

(Side note: I've also seen decision trees, 2, 3, 4, 5, 6, 7 win in several recent Kaggle challenges)


texts: If I want to get how many and what kind of topics are covered by New York Times each week from a bag of words model(All the news covered by NYT in a week) how should I approach? Using traditional unsupervised LDA didn't help much.


texts: How can we perfrom pre fitted standard scalar inverse transform on y variable in pipeline

texts:
I want to create pipeline which will inverse transform the y variable after model prediction
I have 3 pickle files:

X variable scalar transform pre fitted object  :-X sclr
Y variable scalar transform pre fitted object  :-Y sclr
svr model object:-  model

I imported it to python and now I want create pipeline using this file where input data will get scalar transformed by  Xsclr parameter----> then model prediction will happen and then output of model will get inverse transformed  using Y sclr  object parameter
The implementation would look like
from sklearn.pipeline import Pipeline
pipe = Pipeline(steps=[(StandardScaler(), Xsclr),
                       (SVR(), model),
                       (InverseTransformer(), Ysclr)]
)

I actually I wants to convert this pipeline to  ONNX format... Afterwords.