texts:
I'm trying to optimize a neural network architecture for a particular problem, but there just seems to be so many hyperparameters that I'm concerned that there are much better options that I'm not exploring (e.g. I might be getting trapped in a local minima for hyperparams).

Essentially, I'd like some standard bounds for hyperparameter search. Ideally, if a person were to see the breadth of explored parameters, they might be reasonably certain to try another class of machine learning models. In particular, I am looking for concrete advice on at least

What size neurons to include in the search

The number of hidden layers, and the width of the layers
The
optimizer and learning rates
Activation functions and loss functions

Thank you so much!


texts: Orange3 does not show Geo Map tool on OS X

texts:
I am using Orange 3.3.11 and I cannot find "Geo Map" widget on OS X.

But the documentation says there is such a widget. How can I find this widget?


texts: Explanation of random forest performance difference to when using categories and when using dummy variables

texts:
I have some hand coded feature which is a category with values "High", "Low", and "Normal".
I created this feature myself and my problem performance (classification) increased dramatically when using it with expanding these by dummy variables.
Now since I'm trying random forest, I thought I change "High, Low, Normal" to 1, -1, 0 instead.
Now the same model doesn't learn at all.
I thought it should become easier actually for it to split.
Does this have to do with me putting normal to 0?
Thank you for any explanation helping me to understand this.


texts: How to get the relationship/formula between a result and a set of values?

texts:
As the question states I would like to be able to get a formula based on the relationship between potential values (from a set of values) and a result.

Example:

Result: 13

Set of values: 3, 5, 4, 2, 1, 7, 9, 1

I would like to find what equation of the values would give the result (ie. 3 * 4 + 2 - 1), keeping in mind some of the values may be irrelevant in the final equation. The equation would not have to 100% accurate, finding something about 95-98% accurate would be fit for my purpose.

I assume this is a task that regression would be used for however I'm not sure where to begin. Ideally I would like to achieve this in Python or R and I'm guessing there would be some sort of packages that could be leveraged. I have many sets of values and results to run through to hopefully improve the accuracy of the model/equation, I'm not sure how many would be required to get a statistically significant accuracy.


texts: Efficient recurrent network for sequences of varying length

texts:
Suppose I have a bunch of sequences of varying lenghts. The absolute majority of them are short, just a few dozens items long. However, very few of them are significantly longer - more than a hundred items long. The question is, how to organize them efficiently as an input to recurrent layer?

Padding doesn't work, since many sequences need to be heavily padded. Limiting batch size is not an option - these sequences are obtained as parts of a larger structure.


texts: Adaptive regression parameter estimation in R

texts:
How would I do parameter estimation and prediction for the adaptive regression model using R, as in the 4th page of the paper linked below?

http://papers.ssrn.com/sol3/papers.cfm?abstract_id=923635

Could anyone clarify this for me?

If you know adaptive regression models very well, share some useful link, or describe the model/parameter estimation/prediction, that would be very helpful.

Thank you so much!


texts:
def get_pvalue(con_conv, test_conv,con_size,  test_size,):
    lift =  - abs(test_conv - con_conv)
    scale_one = con_conv * (1 - con_conv) * (1 / con_size)
    scale_two = test_conv * (1 - test_conv) * (1 / test_size)
    scale_val = (scale_one + scale_two)**0.5
    p_value = 2 * stats.norm.cdf(lift, loc = 0, scale = scale_val )
    return p_value

I have this function and I would like to know what it is actually doing and how it is actually calculating the p-value.

This is to find the difference between the conversion rate of control and test and group from an A/B test.

con_conv --> Conversion rate for control group
test_conv --> Conversion rate for test group
con_size --> population size for control group
test_size --> population size for test group

I understand that scale_one and scale_two are calculating the variance for each group, but I don't understand why they are adding both of them to calculate the standard deviation and why they are multiplying the cdf with 2 to get the p_value.


texts: Perform fuzzy lookup to partially match the address column

texts:
I have two separate datasets (eg. dataset1.xlsx and dataset2.xlsx). Dataset1 has 2 columns, serial number and service address. Similarly dataset2 has 2 columns service address and customer number. The problem is that the address in both these datasets have spelling errors, for example one of the address in dataset 1 is 790 spring ln, york while the dataset 2 has the same address as 790 spring lane, york. So there is a difference in spelling in both columns. I am trying to match the address in dataset 1 with address in dataset 2. I want the above to addresses to match but because of the spelling error in the word lane it wouldn't match.
The dataset structure is as follows:
DATASET1:
S.NO| SERVICE ADDRESS
1| 101 SUTTON RD,ABBOTTSTOWN
2| 106 E KING ST,ABBOTTSTOWN
3| 430 W KING ST, YORK
4| 130 BEAVER AV,ALIQUIPPA
5| 2601 DUSS AV,AMBRIDGE
DATASET2
S.NO| SERVICE ADDRESS
1| 430 W KING STREET, YORK
2| 2601 DUSS AVENUE,AMBRIDGE
3| 130 BEAVER AV,ALIQUIPPA
4| 106 E KING ST,ABBOTTSTOWN
5| 101 SUTTON RD,ABBOTTSTOWN
When I match the service address columns from dataset 1 and dataset 2, I only get a match between row 4 of dataset 1 and row 3 of dataset 2. I am expecting to get a match between all the rows because they have similar addresses expect for some minute spelling errors.


texts: Inverse predict the features from known target with fitted sklearn regressor

texts:
I understand that the default way a scikit-learn regressor works is that we fit it to a dataset of features and targets (X_train, y_train) and then we can use the fitted regressor to predict an unknown target, y1, from known features, X1.
On the contrary, I know the target, y1, and would like to evaluate what features X1 would yield the known target with the fitted regressor.
I do not think that there is an in-built method (such as inverse_predict) that can achieve this, or is there such a method???
How do you suggest I efficiently estimate X1?
Here is a horribly inefficient way I am doing this presently:
from sklearn.svm import SVR
reg = SVR(kernel='linear')
reg.fit(X_train, y_train)

# While loop to estimate X1
continue_while_loop = True
while continue_while_loop:
    X1 = X_train[np.random.choice(range(X_train.shape[0]),:]  # randomly select a feature vector from X_train
    X1 = X1 + np.random.normal(np.mean(X_train), 10*np.std(X_train), (X_train.shape))  # Add random noise to X1
    y1pred = reg.predict(X1.reshape(1,-1))
    error = np.linalg.norm(y1 - y1pred) # Find the error between predicted and true y1
    if error < 1e-2:
        continue_while_loop = False

print(y1, y1pred)


texts: Unusual Memory trends while using Densenet

texts:
I am using Densenet to train my model but I found some unusual trends of memory usage while training the network, it follows the normal curve. I am unable to understand this can somebody help me explaining this graph.
I am attaching the image of the curve.


texts: Does any cloud service provide free GPU access?

texts: Is there any cloud developer platform which provides free access to an NVIDIA GPU instance, maybe only for a limited trial period? I'd like to work on it before committing to a paid option. I have tried AWS, Google Cloud Developer Platform and IBM Bluemix, but none of them have this option.


texts: Sublime Text 2 with Pandas for Excel (Combining Data) & Data Visualization

texts:
I'm new to programming with Python, and so far it's been a headache to create a build environment- and need your support and expertise in this area.

Background I'm running a MacBook, and using Sublime Text 2, and needing to learn python. I'd like to finish this tutorial on youtube on Data Science (manipulating excel files really "aggregating data on excel"- i'll post the link below and need to import pandas- but apparently I can't.

Also, worth mentioning; I'll also be using Numpy, Xlrd, Matplotlib in the future, and not sure if these modules are available on Sublime Text 2.

The Challenge:

When I run this line, I get an error.

import pandas as pd

I have researched the problem, and it seems that I don't have package control on my sublime text and found this site with the package control addition- so here's the website that I found with a long code to download the package.

https://packagecontrol.io/installation#st2

1) Is this a legit site to download this software?

2) Can I even add these modules to Sublime Text 2 to do Data Science & view excel documents

3) If I add this installation "package control", will this package allow me to download pandas, and other modules into Sublime Text 2

4) If Yes, where can I find instructions for installing the Pandas/Numpy/Xlrd modules?

https://www.youtube.com/watch?v=4_BPNnKEMn8


texts: I am getting an F-Score of 0.99 on the train_test_split data, but only getting 0.40 for a competition's test data. I am oversampling with random forest (after oversampling both of the classes have 95469 data). Before oversampling the data was 0 - 95469 and 1 - 193 respectively.


texts: Can learning algorithms take in data along with their uncertainty? (chaining ML algorithms along with errors)

texts:
How to chain statistical methods (estimators or classifiers) taking into account the uncertainty (error) of the previous step?
Ex: Consider a pipeline, where housing prices are estimated from census and geographical data and are fed into another algorithm to estimate credit scores. How does the error in estimating housing prices fed into credit scores estimator and factor in the overall error?
I think that if you just consider the output values and not the error of the previous step, the error of the current estimator will be lesser and will be misleading. The uncertainty is not being propagated forward in this pipeline, hence the uncertainty at the end is only because of the last step.
I'm new to Machine Learning and in the introductory books or courses I didn't come across any discussion about this topic. If anyone can point me to good resources to learn more about this, I'll be happy.


texts: Where can I find out about the "Helvetica scenario"?

texts:
From the paper introducing GANs:

It makes sense that collapsing too many $\vec{z}$-values to a single $\vec{x}$-value will cause problems. However, I was a bit confused as to how training $G$ for a shorter amount of time before running $D$ can fix it. But when I googled "Helvetica scenario", all I could find were references to a parody show called Look Around You.

Since it seems like this is an actual thing, not just a parody—where can I find out more about it, and how it can be prevented? And why is it called "the Helvetica scenario"?


texts: Detection model - Training with class-instance limit-awareness

texts:
Is there a way to make a detection model aware of the maximal number of possible objects of a given class within a single image?

For example in a toy case with 2 classes. If I know that in every single image, class A can have no more than 5 instances and class B no more than 1. Is there a way to incorporate it into the training process?

To make it clear, I'm not talking about an additional algorithm which runs on top of the trained model (such as non-maximum suppression which is used to select a single bounding box for an object). I specifically ask about the actual model and its training process.


texts: Python to clean miswritten words with repetitive letters such as "wwwwooorrrrddss" to "words"

texts:
When cleaning text-data in Python3 for NLP, are there are any 'common practices' when it comes to addressing repetitive letters in words such as "wwwwooorrds" to "words", or "fffunnnyyyyyy" to "funny"?
The source of the miswritten words is an OCR and I am not able to address the issue upstream, and thought I would check if there was anything that I can do downstream to fix this.
Thanks!


texts: Sequential sampling from Gaussian conditional not working

texts:
I'm trying to sequentially sample from a Gaussian Process prior.
The problem is that the samples eventually converge to zero or diverge to infinity.
I'm using the basic conditionals described e.g. here
Note: the kernel(X,X) function returns the squared exponential kernel with isometric noise.
Here is my code:
n = 32

x_grid = np.linspace(-5,5,n)

x_all = []
y_all = []
for x in x_grid:
    x_all = [x] + x_all
    X = np.array(x_all).reshape(-1, 1)
    # Mean and covariance of the prior
    mu = np.zeros((X.shape), np.float)
    cov = kernel(X, X)
    if len(mu)==1: # first sample is not conditional
        y = np.random.randn()*cov + mu
    else:
        # condition on all previous samples
        u1 = mu[0]
        u2 = mu[1:]
        y2 = np.atleast_2d(np.array(y_all)).T
        C11 = cov[:1,:1] # dependent sample
        C12 = np.atleast_2d(cov[0,1:])
        C21 = np.atleast_2d(cov[1:,0]).T
        C22 = np.atleast_2d(cov[1:, 1:])
        C22_ = la.inv(C22)
        u = u1 + np.dot(C12, np.dot(C22_, (y2 - u2)))
        C22_xC21 = np.dot(C22_, C21)
        C_minus = np.dot(C12, C22_xC21) # this weirdly becomes larger than C!
        C = C11 - C_minus
        y = u + np.random.randn()*C
    y_all = [y.flatten()[0]] + y_all

Here's an example with 32 samples, where it collapses:
enter image description here
Here's an example with 34 samples, where it explodes:
enter image description here
(for this particular kernel, 34 is the number of samples at which (or more) the samples start to diverge.
I've gone through this code so many times that I'm going blind - something must be fundamentally wrong with it but I just can't see it.


texts: What is the underlying difference between linear and non-linear relationship?

texts:
Study on linear correlation coefficient and nonlinear correlation coefficient in mathematical statistics
WANG Ting, ZHANG Shiqiang
Studies in Mathematical Sciences 3 (1), 58-63, 2011
From the above article,I am not clear about the nature of relationship between two variables. The R model - multiple correlation may be working under these concepts.I am not aware of the basis for checking the linear or non-linear relationship with the dependent variable in a multiple regression model?


texts:
I have a question on using sklearn pipelines to predict/classify data.
I understand how to build a pipeline, train it with training data, test it with test data; but after that is where I get lost. How do I use the pipeline to predict values on new/unseen data?

I built a pipeline which transforms categorical data with a OneHotEncoder and another that transforms numerical data by scaling the columns with a StandardScaler. I then used FeatureUnion to combine the two pipelines and join a DecisionTreeClassifier() at the end, to classify the data.
I fit the training data sucessfully with pipeline.fit(X_train,y_train) and thereafter predict with pipeline.predict(X_test). This all works fine.

Next I want to use the pipeline to predict classification of new/unseen data; but when I call pipeline.predict(X_unseen) I get a ValueError. The error relates to one of the categorical features of the data, which is the names of a cities. The pipeline does not seem to transform the unseen data.

Reading through the documentation on pipelines and several examples I understand that when .fit() and .predict is called the data is passed through the entire pipeline. If my understanding is correct then the pipeline.predict(X_unseen) should pass the new data through the pipeline, transforming it and then classifying it. However this does not seem to be the case. Can anyone tell me what I'm missing or misunderstanding?
How do I use the pipeline I built to predict on new data?


texts: How to apply policy gradient to discrete combinatorial action space?

texts:
I'm new to RL techniques, so this may be a dumb question.
It is a combinatorial scenario. For example, the action space is 5*3*4, i.e., I have to make a choice in each of the 3 sub-action sets, which as a whole constitutes the action. I hope the neural network can directly output the 3 sub-actions, perhaps by 3 neurons in the output layer.


texts: What are the advantages of Skip-Gram methods?

texts:
Concerning the notion of word embeddings, Skip-Gram methods aim for computing the probability of a word given its neighborhood. I do not understand the rationale behind it, since it is possible to infer this information by looking directly into the co-occurrence matrix.

In general, I cannot understand those methods aiming to capture as much relevant information from the original co-occurrence matrix as possible. Isn't it easier to work on the co-occurrence matrix directly?

Thank you very much in advance.


texts: What is the best option to test the data set of images using weka?

texts:
I have two hundred and fifty images, and extracted the features from them and put them in an Excel file, how to use the weka program so that the first 200 images for training and the remaining fifty images for testing.

Do I put the data for two hundred images in a separate file and the data for fifty images in another file? Or do I put the data for two hundred and fifty images in the same file?.
Please help me.


texts: Can I use Keras to give fitness scores to a genetic algorithm?

texts: I know that I can use Keras to solve classification problems but is it possible to get it to output a float between 0 and 1000 for instance? I want train my network to approximate the simulation that gives my GA a fitness score because the simulation takes a long time to run.


texts: Gas consumption outliers detection - Neural network project. Bad results

texts:
I tried to detect outliers in the energy gas consumption of some dutch buildings, building a neural network model. I have very bad results, but I can't find the reason.

I am not an expert so I would like to ask you what I can improve and what I'm doing wrong. This is the complete description: https://github.com/denadai2/Gas-consumption-outliers.

The neural network is a FeedFoward Network with Back Propagation. As described here I splitted the dataset in a "small" dataset of 41'000 rows, 9 features and I tried to add more features.

I trained the networks but the results have 14.14 RMSE, so it can't predict so well the gas consumptions, consecutively I can't run a good outlier detection mechanism. I see that in some papers that even if they predict daily or hourly consumption in the electric power, they have errors like MSE = 0.01.

What can I improve? What am I doing wrong? Can you have a look of my description?


texts: SVM with polynomial kernel: strange results with high degree?

texts:
Currently I'm working in WEKA, using the SMO classifier (an implementation of SVM).
For an assignment I am requested to use a polynomial kernel, and report the results for degrees varying from 1 up to 50. I need to comment on overfitting, and my understanding is that higher degrees lead up to a higher risk of overfitting. (And hence, the percentage of correctly classified instances, using 10-fold cross validation should increase as the degree increases).

My intuition is right for the 'first' degrees, resulting in a 99.6% rate at degree=10. However, when I set the degree to 50, WEKA reports a 9% rate.

How is it possible that this rate drops (and by so much)?


texts: Question about Beware of the inherent skew in ranking problems in Google Rule of ML

texts:
I try to understand the Rule 35 from Rule of ML from Google.

  When you switch your ranking algorithm radically enough that different results show up, you have effectively changed the data that your algorithm is going to see in the future. This kind of skew will show up, and you should design your model around it. There are multiple different approaches. These approaches are all ways to favor data that your model has already seen.

Why the algorithm is going to see in the feature? I can't get it.
Is it mean there will be some bias that data is the result from previous model?

  Have higher regularization on features that cover more queries as opposed to those features that are on for only one query. This way, the model will favor features that are specific to one or a few queries over features that generalize to all queries. This approach can help prevent very popular results from leaking into irrelevant queries. Note that this is opposite the more conventional advice of having more regularization on feature columns with more unique values.

It mean put more penalty on those general feature, and less for unique feature. But this opposite the conventional advice, how can we leverage?

  Only allow features to have positive weights. Thus, any good feature will be better than a feature that is "unknown".

What it mean "Unknown"? Negative weights maybe the feature is nagtive correlation?


texts: How to normalize data from multiple sources?

texts:
I am trying to model an individuals' purchasing behavior using different data sources (ex: Zalando, Otto, etc.,). When I combine data sources, I see that the data across these channels is very different.

For example, 5% heavily shop using a particular channel, but do not utilize other channels. When I try to model anything from this information, it performs poorly because overall, its a sparse dataset, but a % of each column should be a very good predictor of a small subset of population.

My question is: How do I combine/normalize such a dataset wherein the data is super sparse?


texts: Getting into data science and NLP outside of university

texts:
I'm almost 40.

Four months ago I resigned from work to rest from stressing work and change my carerr direction. It took me some time to understand what I want to do and only then I started to learn NLP and machine learning through Coursera and Kaggle. To learn new skills and have a minimal portfolio to show during future job interviews,  I've started to work  on a small NLP project. It involves multiple problems like crawling data from the Web, cleaning them, extracting structured data from unstructured text, linking objects based on document similarity, stemming, POS tagging, lemma disambiguation, etc.

However, I feel I'm getting lost and often going into wrong directions. Actually, it starts to look like doing my PhD in the past, reinventing too many wheels. For instance, last week I've spent on porting existing stemmer for my language from Java to Python just to realize a few things. First, that less ideal but simpler solution (Python wrapper) would be enough for my goals and definitely faster to write. Second, I've learned more about software engineering (difference between Java and Python, calling Java from Python) rather than about NLP challenges of the problem (what are the current approaches for stemming, how to train trie-based stemmers, etc.). Third, I've found that the approach I used might be a bit outdated.

I feel lost because of many unknowns:

I don't know if the project that I took makes sense for my future career?
Isn't focusing on my language in my NLP project too constraining? NLP tooling for my language is quite limited when compared to English.
Who knows what are uknown uknowns?

In the end I think it boils down to lack feedback on my progress and actual skills I have vs. what is needed.  The only feedback I've received so far was from one job interview where I've learned that the approach I'm using for my project might be outdated and not necessarily an industry standard. I've looked at the work of a group of students contributing to SpaCy, NLP framework in Python. They have a project supervised by a univeristy teacher or principal, the project requirements have been defined by an external company and they have a chance to work together.

How to get such a feedback when you're no longer a student? Internship? Mentoring? How do you learn alone such things?


texts: Gaussian Process Regression: does feature normalization affect final log-likelihood of model?

texts:
I'm trying to learn a Gaussian Process Regressor in SKLearn.

I tried it both with and without feature (and output) normalization, and even though results seem similar-ish, the reported log marginal likelihood differs a lot (-5000 vs 9000)

My intuition would be that the log likelihood would not be very dependent on feature normalization.

So, my question is: does feature scaling have an effect on the reported log-likelihood?


texts: Is feature scaling necessary in reinforcement learning for the agent to learn successfully?

texts:
I often have trouble deciding how to tweak my input data for the agent. I am changing my data so that values in the hundreds get changed to a value between -1 and 1, and values that are originally small get changed the same ratio. However, my agents often aren't successful.

Could the numeric range of the input data be a cause for the inability to learn?


texts:
For a project I should train a model that predicts whether a certain text given as input (e.g the chapter of a book) is set indoor or outdoor. I do not have any labelled data.
Could you point to relevant work, datasets, algorithm (preferably in python)


texts: Why we use Pickle instead of saving the weights in HDF5 format in Deep Learning

texts:
I have read that, Pickle library is used to save trained model into a .pkl file to use it later.
Also, we can save the weights of a trained model into a hdf5 format using
model.save()

and use those weights afterwards.
So why we use pickle instead of hdf5 files?


texts: Machine Learning - Input Prepocessing - NLP email classification model

texts:
So I created a model which classifies emails into different categories, just like a spam filter. I deployed the model as a webservice, no problem with that but I can’t get my head around how I would use it to predicht the output category of a new email. How do I preprocess the new email (subject and message body) to match the input format of the model/webservice ? The model I trained has about 1000 features, corresponding to the 1000 most frequent words in the training dataset. Do I vectorize the new email ? Do I just search for the features/words in the new email ?
There is something obvious I’m missing, I think.

I used python, sklearn and pandas/numpy to preprocess and train the model


texts:
I am using tensorflow-gpu 1.10.0 and keras-gpu 2.2.4 with a Nvidia gtx765M(2GB) GPU, OS is Win8.1-64 bit- 16GB RAM.

I can train a network with 560x560 pix images and batch-size=1, but after training is over when I try to test/predict I get the following error:

ResourceExhaustedError: OOM when allocating tensor with shape[20,16,560,560] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
     [[Node: conv2d_2/convolution = Conv2D[T=DT_FLOAT, data_format="NCHW", dilations=[1, 1, 1, 1], padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](activation_1/Relu, conv2d_2/kernel/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

I suppose it's a memory issue.

So my question is, is it possible to first use GPU for training and then switching CPU to predict some results in one Jupyter notebook?

Can we free up GPU-memory in windows inside a script?

I found these two topics but I think we should use at the beginning of the script. What I want is switching after training.

https://github.com/keras-team/keras/issues/4613

Switching Keras backend Tensorflow to GPU

Any help will be appreciated,
Thanks


texts: How to implement Multi Layer Perceptron in Keras

texts:
I am trying to implement the network in the paper Two-Stream Deep Feature Modeling for Automated Video Endoscopy Data Analysis available here. However, I am confused about the Multi layer Perceptron in the "Relational Network" part. As far as I can understand from the internet, MLP are fully connected layers are same? Am I correct?
If not, how to implement them in Keras?


texts:
This is the corpus

abcdef
abcd
xabcd
b
bcef
ef
aef

what I want to extract is abcd, ef, then I try to count all n-grams:

abcdef 1
abcd 3
abc 3
...

My question is how to distinguish abcd with abcdef and abc, since abcdef just appear once, and abc is the sub-string of 'abcd'. Of course, I can filter with this two conditions, but just want to know any existing algorithm target on solving this problem.


texts: How to predict auction winning price without knowing the winning price when you lose the auction?

texts:
Suppose I participate in online auctions. I submit bids based on the features of the item being sold, but I only got feedback when I win. If I lose, I would know nothing about who won, and how much was the winning price. I can only assume that you bidded too low.
I would like to use the data of my bidding history, both wins and losses, to predict the winning price and it's winning chance in percentage (the prediction interval). What machine learning problem is this?
I tried building a regression model based on item features using only the winning bid histories, and that worked to some degree. However, I imagine that will be biased, as the losing bid information is not incorporated. For example, if I have the following bid histories:
(3, lose), (4, lose), (5, win), (6, win), (7, win), (8, win)
Using only the winning bid for regression and minimizing for MAE, I would get a price close to 6.5. However, if looking at the full bid history, the actual winning price might be close to 5. So using only the winning price is likely to over predict.
The use case is for real time online advertisement bidding if it's relevant.


texts: Custom Loss Function on a Keras Neural Network

texts:
I'm training a Neural Network on Keras to predict class as a triplet of the form S,P,T, where S, P and T have different values. The output of the Neural Network is one joint triplet.

In my training data, not all entries of the triplet are tagged for all the samples, i.e. for some samples you can find the y value to be s1,_,t1 or _,p2,_ and so on.

I would like to implement a custom loss function that accepts this empty values. I found this example:

def masked_loss_function(y_true, y_pred):
    mask = K.cast(K.not_equal(y_true, mask_value), K.floatx())
    return K.binary_crossentropy(y_true * mask, y_pred * mask)

My problem is: the y_pred and the y_true are one-hot encoded tensors, so before I compare y_pred and y_true I need to decode them and then compare the inner parts of the triplet.

How can I implement this?


texts: How to start analysing and modelling data for an academic project, when not a statistician or data scientist

texts:
I have collected data for a PhD thesis, and need help understanding how to build a road map to do analytical and statistical analysis. The PhD is not itself in statistics or machine learning, but I would like to understand what are the steps and type of analysis that I have to follow for analysing data for an advanced degree? In general, how should I approach such a problem?
In the data I have collected, there are 623 observations including one continuous dependent variable and 13 independent variables (continuous, categorical, and ordinal) that are defined based on the researcher experience and literature review.
I considered planning to do several regression analysis to predict the dependent variable and study the effective factors (if they are positive, negative, and their magnitude) on it. I've tried multiple linear regression including different transformation on independent variables. On the other hand, I'm not sure if I should study each independent variables through the time and  forecast their values in the time horizon?
Here are the steps in my mind so far:

Plotting the scatter plots of different independent variables vs dependent variable to define outliers and check if the model is linear also with respect to coefficients

Removing the potential outliers

Splitting the data into two data sets to build the model and validate it after that.
If the model is linear then:

Performing the multiple linear regression

Performing the multiple linear regression including different transformations to enhance the model

Validating the model

Doing the quantile regression

Doing supervised learning machine etc.

If the model is not linear, I may instead need to use non-linear statistical techniques.
Any feedback would be highly appreciated. My goal is to build a clear and robust road map for this part of the work.


texts: Is it possible to modify or replace the basic network of YOLO?

texts: I have an idea to adapt YOLO algorithm to my application, the original YOLO algorithm is for image classifications, which have 24 convolutional layers with output class of 1000, is it possible to replace the basic network of YOLO with Alexnet or Resnet or the custom network structure designed by myself? Noted that my application have input shape of 500 * 10000 * 1 and only 4 classes for classification.


texts: How to use scikit-learn to extract features from text when I only have positive and unlabeled data?

texts:
I'm looking for something similar to this
https://scikit-learn.org/stable/auto_examples/text/plot_document_classification_20newsgroups.html#sphx-glr-auto-examples-text-plot-document-classification-20newsgroups-py
But instead of positive and negative examples, I have positive examples and a bunch of unlabeled data that will contain some positive examples but is mostly negative.
I'm planning on using this in a pipeline to transform text data into a vector, then feeding it into a classifier using
https://pulearn.github.io/pulearn/doc/pulearn/
The issue is I'm not sure the best way to build the preprocessing stage where I transform the raw text data into a vector which would then be fed into the classification model.
If anyone has any different ideas on how I can transform positive and unlabeled raw text into a vector to feed into the pulearn module I would like to hear as well, thanks!


texts: How to find the size of a tensor in bytes?

texts:
Supposing I have a tensor named encoding which is of the shape (1,32) Now I would like to find out the size of the tensor in bytes not the shape of it. So basically I want to know how much memory it takes.

I found this StackOverflow question which does this exact thing but in TensorFlow 1. So how can I accomplish the same?? Here is the link ===> https://stackoverflow.com/questions/36966316/how-to-get-the-dimensions-of-a-tensor-in-tensorflow-at-graph-construction-time

  The SO question uses the Tensorflow.contrib module which (sadly) has been removed. So anyone know a way?


texts: Importance of Underlying mathematics behind AdaBoost, Gradient Boost and other Machine learning techniques

texts: I am trying to understand how important is it to know the mathematical formula under the hood of these techniques, from interview and job(or practical application) point of view. For example I know how they work, the steps behind everything and how it is computed, like I have proper understanding of Information Gain, Entropy, Gini Index when it comes to decision trees, but I don't know the formula for these. How crucial are they keeping in mind that these are always calculated for us?


texts: Bayesian network in Python: both construction and sampling

texts:
For a project, I need to create synthetic categorical data containing specific dependencies between the attributes. This can be done by sampling from a pre-defined Bayesian Network. After some exploration on the internet, I found that Pomegranate is a good package for Bayesian Networks, however - as far as I'm concerned - it seems unpossible to sample from such a pre-defined Bayesian Network. As an example, model.sample() raises a NotImplementedError (despite this solution says so).

Does anyone know if there exists a library which provides a good interface for the construction and sampling of/from a Bayesian network?


texts:
There are several possibilites to evaluate a model:

hist = model.fit(x_train, y_train,
          (...)
          validation_data=(x_test, y_test))

or to use .evaluate() resp. .predict(). As far as I know I would use X_test and y_test for validation and prediction.

My question: How are they different? When I .fit() and .evaluate() afterwards, which information do I gain by .fit(validation_data(...))?


texts: Clustering together words that appear together while down weighting words that appear too often

texts:
I was wondering if I could get some help finding a good model for the problem I have. I have a data set where each observation is a set of words that go together. So for example, it could be:
Obseration 1: {car, pizza, exhaust, engine}
Observation 2: {car, pizza, engine}
Observation 3: {food, pizza, chips}
.
.
.
Observation x: {ballons, air, pizza}
Observation x + 1: {car, exhaust}
I am trying to find a model that, when given a word (for example, "car"), it returns the words that are most commonly used with that word. One way to do this is to use cosine similarity however, there is an additional constraint I am trying to handle. In the example above, the word "pizza" is a word that's super common in a lot of observations. The thing is that, because it's common with so many observations and topics, I don't want to include it as one of the synonymous words that get shown or, at the very least, decrease the probability that it gets selected via some sort of weighting method.
Essentially, I am looking for words that go together often with each other but don't go together universally with many other words (if that makes sense).
Any models you guys have in mind? I would really appreciate the help! I have thought of doing something like a zipf's law weighting of some sort where the most common words are down weighted based inversely on how frequently they show up but wondering if there are some machine learning methods that are just built for this already!
Thank you for any responses!


texts: How many coefficients does the Logistic regression model has as a function of the number of features?

texts:
I have built a logistic regression model using Python anaconda and was surprised to see that the number of model coefficients turned out to be proportional to the training sample size i.e.

My training data is:

print('Training data is type %s and shape %s' % (type(os_X_train), os_X_train.shape))

and outputs:

Training data is type <class 'pandas.core.frame.DataFrame'> and shape (174146, 11)

Then the model is:

logreg = LogisticRegression(penalty='l2',solver='lbfgs',max_iter=1000)
model = make_pipeline(preprocess, logreg)
model.fit(os_X_train, os_y_train)
print(logreg.coef_.shape)
print("Model coefficients: ", logreg.intercept_, logreg.coef_)

This outputs:

(1, 153024)
Model coefficients:  [12.02830778] [[ 0.42926969  0.14192505 -1.89354062 ...  0.008847    0.00884372 -8.15123962]]

To my understanding the number of model coefficients should be the number of columns for the predictor variables or features plus one the intercept, or?


texts: Handling a feature containing multiple values

texts:
I have a dataset in following format:

Movie ID |  Actors | Director | language | ReleaseYear | Genre

1  |    Anil Kapoor;Manisha Koirala;Jackie Shroff;Anupam Kher;Danny Denzongpa;Pran  | Vidhu Vinod Chopra  | hin | 1994  |  Drama;Romance;Patriotic

As you can see the columns Actors and Genre has multiple values. I need to perform cluster analysis on this dataset. I don't know what would the best way to deal with such data. I am thinking of two possible solution(don't know if it is a correct approach for this type of problem)--

Split the column, say, 'Actors' into multiple columns e.g. Actors 1, Actor2,..., and then perform the cluster anylysis.
Row split i.e. convert a single row of data into multiple rows each row for one value of columns like 'Actors' and 'Genre'.

Please suggest me the best way to deal with such data for cluster analysis.


texts: Quantifying binary classification uncertainty

texts:
I am designing a binary classifier and would like some ideas for how best to quantify uncertainty in the final model classification. The design process consists of first training the model, then performing ROC analysis, and finally choosing a threshold such that the probability of false alarm is ~0.01. So, when an unknown sample is passed into this classifier, it outputs a number between 0 and 1, and if the number is above the threshold, the sample is classified into class 1 (C1), or into C0 if the number is below the threshold.
My question is about how to design a confidence score around the final classification. Say my threshold is 0.9. So, if the model outputs a score of 0.91, then my sample is classified as C1, but if the output score is 0.89, then the sample is classified as C0. Intuitively, given each of these scores, I should have lower confidence in the labels than if the model output was 0.99 or 0.01, respectively. One possibility is to simply compute a normalized [0,1] distance from the score to the threshold, like: $|\hat{y} - T|/(1 - T)$ for predictions above the threshold (i.e., samples labeled as C1), and $|\hat{y} - T|/T$ for predictions below the threshold (where $\hat{y}$ is the model output and $T$ is the threshold). One issue with this is that it's not symmetric. That is, for a threshold of 0.9, a model output of 0.91 has an "certainty score" (i.e. the computation shown above) of 0.1, while a model output of 0.89 has an certainty score of 0.011.
Another possibility that occurs to me is the probability of missed detection or the probability of false alarm for instances where $\hat{y}<T$ (sample classified as C0) or $\hat{y}>T$ (sample classified as C1), respectively. So, for a model output of 0.89, I would compute $FNR = 1- sensitivity$, while for a model output of 0.91, I would compute $FPR = 1-specificity$, and use these as measures of uncertainty in the final classification.
Are there other suggested methods for computing this kind of classifier uncertainty?


texts:
I am preparing a dataset for a model, but somehow the code just doesn't run well.

The major error is:

File "/Users/liangjulia/Desktop/UW DS Certificate Learning Material/untitled6.py", line 61
  'income2' = pd.to_numeric(Adult.income, errors='coerce')
                                                        ^
SyntaxError: can't assign to literal

Code:

# import statement
import numpy as np
import pandas as pd
import csv
import matplotlib.pyplot as plt
from sklearn.preprocessing import LabelEncoder
# loading dataset, it is a combination of categorical and numerical data
hp = pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data",header=0,sep=',')
hp.columns = ['age','workclass','income','education','education-num','marital-status','occupation','relationship','race','sex','capital-gain','capital-loss','hours-per-week','native-country','salary-range']
# dataset basics
hp.head()
hp.shape
hp.dtypes

# account for all value '?'
hp.replace('?','na')
hp.isnull().sum()

#Remove onsolete data point in income
hp('income').dropna()

# replace all aberrant values
hp.replace('nan', 0)
hp.replace('NULL', 0)

# change data type of certain data point to numerical
number = LabelEncoder()
hp['income'] = number.fit_transform(hp['income'.astype('str')])
hp['capital-gain'] = number.fit_transform(hp['capital-gain'.astype('str')])
hp['capital-loss'] = number.fit_transform(hp['capital-loss'.astype('str')])

# Choose the datapoint 'income' to perform the data cleaning and remove outliers
LimitHi=np.mean('income') + 2*np.std('income')
LimitLo=np.mean('income') + 2*np.std('income')
BadIncome = ('income' > LimitHi) & ('income' < LimitLo)

# Replace outliars
RightIncome = ~BadIncome
x[BadIncome] = np.mean(x[RightIncome])

# normalize the Income Column using numpy
#'income2' = pd.to_numeric(Adult.income, errors='coerce')
minmaxscaled =('income' - min('income'))/(max('income') - min('income'))

# bin age data into several ranges
hp['bin'] = pd.cut(hp['age'], [15,30,45,60,75,90])

# construct new categorical data point with existing data point
hp['EvalonInvestment'] = 'zzz'
hp.loc[(hp['capital-gain'] >= 50000), 'loc2'] = 'investmentking'
hp.loc[(hp['capital-gain'] > 10000) & (hp['capital-gain'] < 50000), 'loc2'] = 'good-investment'
hp.loc[(hp['capital-gain'] > 0) & (hp['capital-gain'] <= 10000), 'loc2'] = 'ok-investment'

print(hp)


texts: Object detection/recognition, pre-processing error

texts:
My Imports:
# Importing modules
import numpy as np
import pandas as pd
import os
import matplotlib.pyplot as plt
import cv2

from keras.utils import to_categorical
from keras.layers import Dense,Conv2D,Flatten,MaxPool2D,Dropout
from keras.models import Sequential

from sklearn.model_selection import train_test_split

My Code:
np.random.seed(1)

train_images = []
train_labels = []

shape = (108,108)

label_path = 'Beer/ModelSet/'

train_labels.append('miller_lite')
train_labels.append('stella_artois')
train_labels.append('michelob_ultra')
train_labels.append('belgian_blue')

for folder in os.listdir(label_path):
    for files in os.listdir(label_path+folder):
        img = cv2.imread(os.path.join(label_path,folder,files))
        train_images.append(img)

train_labels = np.asarray(pd.get_dummies(train_labels).values)

train_images = np.asarray(train_images)

x_train, x_val, y_train, y_val = train_test_split(train_images, train_labels, random_state=1)

The part that fails
    x_train, x_val, y_train, y_val = train_test_split(train_images, train_labels,
random_state=1)

The reason it fails
ValueError: Found input variables with inconsistent numbers of samples: [20000, 4]

My Error:
2020-08-03 23:47:11.117431: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Traceback (most recent call last):
  File "/Attempt-2.py", line 40, in <module>
    x_train, x_val, y_train, y_val = train_test_split(train_images, train_labels, random_state=1)
  File "python3.8/site-packages/sklearn/model_selection/_split.py", line 2127, in train_test_split
    arrays = indexable(*arrays)
  File "python3.8/site-packages/sklearn/utils/validation.py", line 293, in indexable
    check_consistent_length(*result)
  File "python3.8/site-packages/sklearn/utils/validation.py", line 256, in check_consistent_length
    raise ValueError("Found input variables with inconsistent numbers of"
ValueError: Found input variables with inconsistent numbers of samples: [20000, 4]

The Goal:
Create a classification model that distinguishes between 4 different brands of beer bottles. MillerLite,StellaArtois,MichelobUltra, and BelgianBlue.
Background Note:
I have never had any practical/work experience with either software engineering, data science, software development, machine learning etc. I am just a student messing around for my own amusement/fun.
The Question:
How exactly do I fix this? I understand the problem is that matrix x is not the same size as matrix y. X is 20000 images of size 108,108 and 3 channels RGB. Y is the label matrix:
[[1 0 0 0][0 1 0 0][0 0 1 0][0 0 0 1]]
in order to split into train/test images the error says I need to have the same len/size array/matrix for both x and y.


texts: Iterative Reweighted Least Squares in python

texts:
I am trying to manually implement the irls logistic regression (Chapter 4.3.3 in Bishop - Pattern Recognition And Machine Learning) in python.
For updating the weights, I am using $w' = w-(\Phi^TR\Phi)^{-1}\Phi^T(y-t)$
However I am not getting satisfying results, also my weights are growing unbounded in each iteration.

I've written this code so far:

def y(X, w):
    return sigmoid(X.dot(w))

def R(y):
    R = np.identity(y.size)
    R = R*(y*(1-y))
    return R

def irls(X, t):
    w = np.ones(X.shape[1])
    w = w.reshape(w.size, 1)
    t = np.array(list(map(lambda x: 1 if x else 0, t)))
    t = t.reshape(t.size, 1)

    #after 3 iterations matrix is singular
    for i in range(3):
        y_ = y(X,w)
        w = w - np.linalg.inv(X.T.dot(R(y_)).dot(X)).dot((X.T).dot(y_-t))
    return w

where X is my design matrix and t is my target vector comprising Boolean values (data is from https://archive.ics.uci.edu/ml/datasets/Mice+Protein+Expression).

Any help is appreciated pointing out where I went wrong.


texts: Error while trying to do hyperparameter tuning using hyperas

texts:
I am getting a syntax error while using hyperas and am not sure why.

My code:

import hyperas, hyperopt

from tensorflow.keras.layers import Dense, Dropout, Activation, Input
from tensorflow.keras import regularizers
from tensorflow.keras.models import Model

from hyperas import optim
from hyperas.distributions import choice, uniform

from hyperopt import Trials, STATUS_OK, tpe
#trials - database to store all intermediate calculations / point evaluations of the search
#status_ok returns ok if search operation is successfully completed and returns fail if the fucntion is not defined.
#tpe - bayesian hyperparameter tuning.
def data_(X_train,X_val):
    return( X_train,X_train,X_val,X_val)

def model_(X_train, X_val):
    input_dim = X_train.shape[1]
    encoding_dim=[{{choice([6,8])}}]
    input_layer = Input(shape=(input_dim, ))
    encoder = Dense(encoding_dim, activation={{choice(['sigmoid','tanh'])}},
                    activity_regularizer=regularizers.l1({{choice([10e-5,10e-4,10e-3])}}))(input_layer)
    encoder = Dense(int(encoding_dim / 2), activation={{choice(['leaky_relu','relu'])}})(encoder)
    decoder = Dense(int(encoding_dim / 2), activation=({{choice(['sigmoid','tanh'])}}))(encoder)
    decoder = Dense(input_dim, activation=({{choice(['leaky_relu','relu'])}}))(decoder)
    autoencoder = Model(inputs=input_layer, outputs=decoder)
    nb_epoch={{choice([100,150,200])}}
    batch_size={{choice([512,1024])}}
    #adam=keras.optimizers.Adam(lr={{choice([10**-3,10**-2,10**-1])}})
    #rmsprop=keras.optimizers.RMSprop(lr={{choice([10**-3,10**-2,10**-1])}})
    #sgd=keras.optimizers.SGD(lr={{choice([10**-3,10**-2,10**-1])}})
    autoencoder.compile(optimizer='sgd',
                    loss='mean_squared_error',
                    metrics=['accuracy'])
    history = autoencoder.fit(X_train, X_train,
                    epochs=nb_epoch,
                    batch_size=batch_size,
                    shuffle=True,
                    validation_data=(X_val, X_val),
                    verbose=1).history
    print('acc',history['accuracy'])
    print('val_acc',history['val_accuracy'])
    return({'acc':history['accuracy'],'val_acc':history['val_accuracy'],'status':STATUS_OK,'model':history})

best_run,best_model=optim.minimize(model=model_,data=data_(X_train,X_val),algo=tpe.suggest,max_evals=5,trials=Trials(),notebook_name='autoencoder')

I got the following error:

Traceback (most recent call last):

      File "/usr/local/lib/python3.5/dist-packages/IPython/core/interactiveshell.py", line 3291, in run_code
        exec(code_obj, self.user_global_ns, self.user_ns)

      File "<ipython-input-148-fcddad2ebef6>", line 2, in <module>
        notebook_name='autoencoder')

      File "/usr/local/lib/python3.5/dist-packages/hyperas/optim.py", line 69, in minimize
        keep_temp=keep_temp)

      File "/usr/local/lib/python3.5/dist-packages/hyperas/optim.py", line 98, in base_minimizer
        model_str = get_hyperopt_model_string(model, data, functions, notebook_name, verbose, stack)

      File "/usr/local/lib/python3.5/dist-packages/hyperas/optim.py", line 189, in get_hyperopt_model_string
        imports = extract_imports(cleaned_source, verbose)

      File "/usr/local/lib/python3.5/dist-packages/hyperas/utils.py", line 40, in extract_imports
        tree = ast.parse(source)

      File "/usr/lib/python3.5/ast.py", line 35, in parse
        return compile(source, filename, mode, PyCF_ONLY_AST)

      File "<unknown>", line 520
        kmeans=
               ^
    SyntaxError: invalid syntax


texts:
I'm trying to run language detection using Facebook's fastText through a Python script but I get this error when I load the model :

  Exception: fastText: Cannot load lid.176.bin due to C++ extension failed to allocate the memory

Any ideas? I've already installed all prerequisites and still get this error.


texts:
There's this side project I'm working on where I need to structure a solution to the following problem.

I have two groups of people (clients). Group "A" intends to buy and group "B" intends to sell a determined product "X".

The product has a series of attributes x_i and my objective is to facilitate the transaction between "A" e "B" by matching their preferences. The main idea is to point out to each member of "A" a corresponding in "B" who’s product better suits his needs, and vice versa.

Some complicating aspects of the problem:

The list of attributes is not finite. The buyer might be interested in a very particular characteristic or some kind of design which is rare among the population and I can’t predict. Can’t previously list all the attributes;
Attributes might be continuous, binary or non-quantifiable (ex: price, functionality, design).

Any suggestion on how to approach this problem and solve it in an automated way?
The idea is to really think out of the box here so feel free to "go wild" on your suggestions.

I would also appreciate some references to other similar problems if possible.


texts: Chinese restaurant process vs Dirichlet Process

texts:
On Wikipedia Dirichlet Process page, regarding the connection between the Chinese restaurant process and the Dirichlet process it's state the following

  If one associates draws from the base measure
  H with every table, the resulting distribution over the sample space S is a random sample of a Dirichlet process.

What does it mean to:

  Associate draws from the base measure H with every table?

It doesn't make any sense to me.


texts: How would I chi-squared test these simple results from A/B experiment?

texts:
I have results from an A/B experiment where users could do one of three things: Watch, Interact, or Nothing

My data is like this:

   Watch  | Nothing | Interact
A: 327445 | 271602  | 744702
B: 376455 | 140737  | 818204

I tried to use the chisquared test bundled with scipy. I'm completely new to data science, but I believe this is the evaluation metric I would want.

scipy.stats.chisquare([ [327445, 271602, 744702], [376455, 140737, 818204] ])
Power_divergenceResult(statistic=array([  3412.38826538,  41532.93339946,   3456.72996585]), pvalue=array([ 0.,  0.,  0.]))

This does not look like a valid result...

I even tried adding an expected frequencies options, but without success. Maybe I'm missing something, either about evaluating this type of data or just not using scipy correctly. Can anyone help me?


texts: How to make a decision tree with both continuous and categorical variables in the dataset?

texts:
Let's say I have 3 categorical and 2 continuous attributes in a dataset. How do I build a decision tree using these 5 variables?
Edit:
For categorical variables, it is easy to say that we will split them just by {yes/no} and calculate the total gini gain, but my doubt tends to be primarily with the continuous attributes. Let's say I have values for a continuous attribute like {1,2,3,4,5}. What will be my split point choices? Will they be checked at every data point like {<1,>=1......& so on till} or will the splitting point will be something like the mean of column?


texts: Generating a Confusion Matrix from a Seaborn Heatmap

texts: Is there a way I can reverse engineer a Seaborn heatmap saved as .png? I want to get the array of values out of it. Does the library provide a "load-from-image" facility, or is there any sta


texts: Which algorithm to use for predicting late deliveries at warehouse?

texts:
I am trying to predict risk labels of late deliveries for the upcoming 1-2 months based on 3 years of historic delivery data.
The target variable is a categorical output based on how many days late a delivery was (on time, 1-2 days late, one week late, more than one week late).
Input variables include things like material information, suppliers location & distance to warehouse, # of times delivery was marked as delayed and suppliers monthly quality ratings as well as information on known quality issues registered with supplier over time etc.

So looking at the information above and that I have records per year of around 500.000 delivieries from 300 suppliers I believe random forests would be a good choice to start.

However, I´m thinking it would not capture important time sensitive trends, such as a supplier all of the sudden delivering late while he was reliable the last 3 years.
So what is a good way to capture time sensitive trends in RF or should I choose a different algorithm form the time-series domain?
Or is there a way to combine multiple approaches?

Thanks for you input!


texts: Comparing functional hypotheses accounting for uncertain interpretation of their predictions

texts:
I am interested in using an information-theoretic approach (likely AIC) to compare the explanatory power of several functional hypotheses. As an example, hypothesis H1 predicts significant association between the dependent variable X and one or more of the variables A, B, C, while hypothesis H2 predicts significant associations between X and one or both of the variables E, F.
Though I may anticipate that D and E correlate with X under H2, due to a lack of prior work on the system this is not certain. This is to say, finding that D is significantly associated with X and that E is not, or vice versa, should not be seen as weaker support for H2 than if both variables were significantly associated with X. Said a third way, I don’t know if it’s appropriate to potentially penalize a model for my imprecise interpretation of its predictions.
Given this is, it best to:
(1) formulate each hypothesis as a full model, i.e. H1=X~A+B+C and H2=X~E+F or
(2) formulate each hypothesis as every possible sub-set of associated variables, i.e. H11=X~A+B+C, H12=X~A+B, H13=X~A+C ... H1k=X~A or perhaps
(3) perform feature selection on each hypothesis to determine the best set of variables to characterize that hypothesis?


texts: Should you use the same algorithm in your feature selection as your model

texts:
My question is should you use the same algorithm in feature selection as your model?
If I'm using a KNN model for classification should I also use a KNN algo when running feature selection? Or lightGBM algo for feature selection and lightGBM model?
Or is there a best algo for feature selection that you can use those features with any model?
I know there are several different feature engineering techniques I just don't know which techniques should be used with different machine learning algorithms.


texts: How to plot a world map with cities in the right locations?

texts:
The data source is a CSV with cities in various countries in the world, each city has some numeric data associated with it. df.columns looks like:

['Continent', 'Country', 'City', 'data']

I want to make a map with all these cities, shown in the right location. I want a color bar with at least some of the cities ordered by the numeric data column. On the map, the cities should be marked with the same color as in the color bar.

Probably something like Plotly could draw the map. But:

how to get the geolocations of each city based on the first 3 columns?
how to plot dots or disks (with color attributes) in Plotly based on lat/long pairs?

The idea is similar to this, but with cities instead of countries:


texts: Are non-relu activations better for small/ dense datasets?

texts:
Building on the questions below, the only conclusion I could draw from the answers was that ReLu is less computationally expensive and better at sparsity.

Why is ReLU used as an activation function?
Why ReLU is better than the other activation functions

Am I better off using a smoother activation like tanh if I have a smaller, dense dataset?


texts:
Looking at the first paper on RDDs/Apache Spark, I found a statement saying that "RDDs degrade gracefully when there is not enough memory to store them, as long as they are only being used in scan-based operations"

What are scan-based operations in the context of RDDs and which of the Transformations in Spark are scan-based operations


texts: How would I implement a model in Keras to generate embeddings based on documents?

texts: I would like to create a model which generates embeddings for documents. I can create the model, but I was wondering how I would implement a training scheme where data consisted of two documents, either identical or different, and would be fed in parallel to the model (two instances running at the same time), which would create embeddings for their respective document. I want to use cosine distance to see similarity, and distance would be lower for documents that are the same. How would I train/implement this?


texts: What are the tools to speed up the running time of machine learning algorithms?

texts:
I know Spark as the fastest tools for Data processing, but not sure if it would be useful to speed up the running time of ML algorithms. For example, my ML models to be built needs about 24h when I run on my machine on RStudio. I am thinking another tool to reduce this expensive time-cost.

I am talking mainly about the distributed computing tools like spark regardless of the type of algorithm and the size of my data.

Any help is appreciated !


texts: Has a past data outside sliding window any Influence on learning of a LSTM?

texts: I am getting into LSTM  and I have one technical question. I have time  series in 1 minute interval and 2 weeks data.  I need predict every next minute in the future. One sliding window has 120 minutes length. My question is, if " the past data outside  of one sliding window" has an influence on training and testing in current window of a LSTM ?  Different way  - could be  the sliding windows from the 2 weeks mixed randomly for training ( out of any order) ? Thank`s.


texts: What kind of machine learning can predict a probability rather than classification?

texts:
For example, I want to predict a probability of whether raining tomorrow, and I can't get the prob. of cloudy or sunny...etc day.

I only get events(attributes of weather) like temperature, humidity, sunny interval, rainy interval...etc.

Train data:
temp. -> 80F
humidity -> 30%
sunny interval -> 5hr
rainy interval -> 0hr

Label: prob. of rainy tomorrow is 20%

Test data example:
temp. -> 90F
humidity -> 10%
sunny interval -> 10hr
rainy interval -> 0hr

I only can get the prob. of rainy day, so I can't use classification algo.

Like above, I want to predict the prob. of rainy tomorrow. I'm not a professional in ML knowledge. I survey many sources and thought the most possible algorithm of ML maybe is regression!?(Cuz it can give me a value like linear regression).

But I don't know is there any other algorithms can help me to predict a probability(also can say I want to predict a value)?


texts: Is Gini coefficient a good metric for measuring predictive model performance on highly imbalanced data

texts:
I am evaluating a Credit Risk model that predicts the estimated likelihood of customers defaulting on their mortgage accounts. The model is a Logistic Regression estimator and was built by another team. They use the Gini metric to measure the performance of the model. They achieved 87%. Upon evaluation, I found that the recall was 51% whilst the error rate of the non rare event class (do not default) was 0.9%. Am I correct in thinking that the Gini is actually a misleading metric in this case because it doesn't really show the extremely poor predictive performance of the rare event class? I have questioned them about this and tried to recommend them to use precision/recall metrics as well as confusion matrices and a precision-recall trade-off graph but they quickly dismissed me.

Any advice would be much appreciated.


texts:
Are Question answering (QA) the same as Chatbots? I can not understand the difference between them.

For me it's the same thing: interact with a robot that answers questions.


texts: Test Binary Classifier on a Test-set that includes only one class

texts:
I'm working on a disease binary classification problem.
0 = healthy , 1 = not healthy
The disease is a movement disorder that appears on the patient while moving a specific movement.
I applied leave-one-out cross-validation to train on all patients except one to test on and so on.
The problem is, some patients don't show class not healthy at all in their datasets. Thus, the evaluation metrics f1-score, precision & recall went down sharply because of those patients.
I tried to use SMOTE oversampling, but it didn't work because it will generate new samples from the data of other patients. Additionally, I tried class-weights, but it didn't work as well, because there is not a minor class in the test set to give it a higher priority.
How can I solve this problem?


texts: How to draw line plot with multiple lines on Tableau?

texts: I have 4 different measures which I want all on the SAME line plot. They do have the same scale. I tried dual axis but it doesn't work for more than 2 plots.


texts: What is difference between Multi-class One vs All and Multilabel Classification?

texts:
Although Multi class is different from Multi label classification, whats difference does adding One vs All make in Multi-class.

Edit 1:
http://scikit-learn.org/stable/modules/multiclass.html#multilabel-learning

In this Link the part where it mentions supported classifiers under bullets of Multiclass as One-Vs-All and Support Multilabel is confusing.


texts: Query similarity: how much data is used in practice?

texts:
I recently read Similarity Measures for Short Segments of Text (Metzler et al.).  It describes basic methods for measuring query similarity, and in the paper, the data consists of queries and their top results. Results are lists of page urls, page titles, and short page snippets.  In the paper, the authors collect 200 results per query.

When using the public Google APIs to retrieve results, I was only able to collect 4-10 results per query.  There's a substantial difference between 10 and 200.  Hence, how much data is commonly used in practice to measure query similarity (e.g., how many results per query)?

References are a plus!


texts: Unsupervised encoding of categorical features

texts:
I have multiple log records with discrete categorical features. Shape of my dataset is (100k, 24) My aim is to look for anomalies in these records. I am planning to cluster the data after encoding.

Before going forward with any analysis, the categorical features need to be encoded; I'd have ideally gone with LabelEncoder or OneHotEncoder but the issue is some features have > 40 possible values.

As of now, I've frequency encoded the variables but is there something I am missing and is there a better way of going about this?


texts: Does TargetEncoder class from category_encoders include cross validation

texts:
I learnt (from below links) that to effectively do target encoding without overfitting, we have to do cross validation for each fold (so kind of double validation) and compute the encoding values of categories based on the mean values of the remaining folds.

IS this double cross validation already included in the 'TargetEncoder' class in categorical_encoders library ?
If I use the targetencoder directly in a pipeline without any extra cross validation, will it be valid (like the double cross validation) or will it lead to overfitting?

From the below implementation on github of the class, I don't see any CV being done in fit and transform.
https://github.com/scikit-learn-contrib/category_encoders/blob/master/category_encoders/target_encoder.py
Encoding categorical variables using likelihood estimation
Target encoding with cross validation


texts: How to force weights to be non-negative in Linear regression

texts:
I am using a standard linear regression using scikit-learn in python.
However, I would like to force the weights to be all non-negative for every feature. is there any way I can accomplish that? I was looking in the documentation but could not find a way to accomplish that.


texts: A the end of a big DS project, should I make trained models available on GitHub?

texts:
I almost completed two big Data Science personal projects based on Deep Learning. They are the fanciest models I've implemented up to now, and I'm pushing all my code on GitHub.

Do you advice to upload trained models too? Or should I let other users run the code and get their own?

What do you do? What are pros and cons?


texts: xgboostclassifier prediction error after saving the model and restoring it

texts:
I have trained a xgboost model and during training, the prediction works fine. But if I stop the script and start a restoring script to restore and predict, then for the same test dataset I get every data classified into one class. The weird part is that, even with the first prediction, i restore the model as shown below,

X = df.drop(['label'], axis=1)
y = df['label']
training_count = 0

X_train, test_data, y_train, test_label = train_test_split(X, y, test_size=0.1, random_state=7)
model = XGBClassifier(learning_rate=0.5, n_estimators=250, max_depth= 5)
model.fit(X_train, y_train)
model.save_model('trained_model_full')

#validation

model = XGBClassifier(learning_rate=0.5, n_estimators=250, max_depth= 5)
booster = xgb.Booster()
booster.load_model('trained_model_full')
model._Booster = booster
model._le = LabelEncoder().fit(test_label)
start = time.time()
pred = model.predict(test_data)
end = time.time()

The above code works and gives me 99% accuracy. But if I remove the training part and just restore it like below, then it fails to work. I get 50% accuracy.

X = df.drop(['label'], axis=1)
y = df['label']
training_count = 0

X_train, test_data, y_train, test_label = train_test_split(X, y, test_size=0.1, random_state=7)

#validation

model = XGBClassifier(learning_rate=0.5, n_estimators=250, max_depth= 5)
booster = xgb.Booster()
booster.load_model('trained_model_full')
model._Booster = booster
model._le = LabelEncoder().fit(test_label)
start = time.time()
pred = model.predict(test_data)
end = time.time()

This is a strange issue. Have anyone come across something like this? if so could you help me out?


texts: Understand the shape of this Convolutional Neural Network

texts:
I'm trying to implement what is explained in a paper on audio signal processing. The guys who wrote this paper tried a Convolutional Neural Network and here is how they explain it :

"The CNN are constructed using two different building blocks: block A consists of two layers with 32 3*3 filters, and block B consists of two layers with 64 3*3 filters ; both in combination with batch normalization, and each followed by a 3*3 max-pooling layer and a drop-out layer."

see the full paper here

They state that they apply this model to 84*9 images (actually it's data from a spectogram, 84 is a number of frequency bins, 9 is the number of frames)

I have some trouble to find out precisely the different layers to add to the model :

Does the Batch Normalization layer occurs after each convolutional layer ?
Does the max-pooling layer occurs after each convolutional layer ?
Does the drop-out layer occurs after each convolutional layer ?
Is there any implicit padding ? Implicit stride ?

My guess is : if they say they are using Batch Normalization, max-pooling and drop-out, then maybe it is implemented after each Convolutional layer (the four of them) (I've often seen that it's applied after each layer)
But with those layers it's impossible to have an input_shape of (84, 9) with no padding ...
I'm quite stuck.

Any ideas ?

Thanks a lot :)


texts: Create a new column in Pandas Dataframe based on the 'NaN' values in another column

texts:
What is the most efficient way to create a new column based off of nan values in a separate column (considering the dataframe is very large)

1 2 3
4 5 NaN
7 8 9
3 2 NaN
5 6 NaN

Should give

1 2 3    0
4 5 NaN  1
7 8 9    0
3 2 NaN  1
5 6 NaN  1

EDIT:

What if it were based on 2 columns? Like:

1 2   3    0
4 NaN 1    1
7 8   9    0
3 2   NaN  1
5 NaN 2    1


texts: Convolutional neural network for gray images

texts:
I am using vgg16 to design a CNN that takes gray input images. The model give me good results without changing anything related to colors.

I am not sure if what I did is correct or not. I want to know should I change the model to be suitable for gray images?
Is what I did is correct?


texts: How to represent time based periodic data for use in ML

texts:
I have a dataset of following format -

Number of Machines
CustId   Month0   Month-1   Month-2   Month-3   Month-4
abc      23       26        29        0         0
def      53       26        22        22        12
ghi      11       26        150       120       10

Size of data protected
CustId   Month0   Month-1   Month-2   Month-3   Month-4
abc      23       26        29        0         0
def      53       26        22        22        12
ghi      11       26        150       120       10

The data is Month-over-Month data. For simplicity I have used the same data in both tables, but for a given CustId data will be present in both the tables. Similarly there are tables for other parameters as well.

I want to use machine learning for some classification. What is the best way to serialize this MoM data for different parameters? Is there any standard practice for this?

Edit:

I am not sure of what are the distinctive patterns in the graphs of the parameters for different classification categories. By graph of the parameter I mean Number of machines plotted against Month.

One thing I can do is to combine all the tables with format something like -

    CustId   ----# machine---       -------data protected-----
             Month0   Month-1 ...   Month0   Month-1   Month-2 ...
    abc      23       26      ...   23       26        29      ...
    def      53       26      ...   53       26        22      ...
    ghi      11       26      ...   11       26        150     ...

But I don't want to use this format because in here all the columns (monthly data) would be treated as independent data points. I want the columns under '# machine" category to be treated as group and the columns under "data protected" as another group.

I came across this Python Library (tsFresh) which extracts information from the graphs. I am looking for something similar or any other way to approach this problem.


texts: Hadoop - checksum while reading file from client

texts:
In hadoop , if i want to read a file from client , the class DFSInputStream enters in action.

About this, my textbook says

The DFSInputStream also verifies checksums for the data transferred to it from the datanode.
If a corrupted block is found, the DFSInputStream attempts to read a replica of the block from another datanode; it also reports the corrupted block to the namenode.

My questions are :
1) What's the meaning in saying that the data is trasferred to DFSInputStream ?
 Am i wrong, or the client communicates directly with the datanode ?

2)If the checksum shows that the transfer of a block has going wrong, why does the DFSInputStream request this block from another datanode instead of trying again with the datanode that generated the corrupted block ?


texts:
If I come across decision trees, it is a binary tree with predicates internal nodes.
How often do we use m-ary decision trees?
Is there any combination of m-ary and binary decision tree, e.g. first level of the tree is binary and second level of the tree consists of m-ary?
Same question arises for Random Forest.


texts: Predict how many days until New Purchase

texts:
I am trying to introduce myself into predicting with dates and I am having big troubles understanding how to do it.
I have a dataset with customer orders from an e-commerce platform with the usual variables (user_id, gender, amount of purchase, date of purchase, country, etc), and I would like to predict, based on historical data, how many days will take a customer to buy a second time, given that this customer already bought before.
So far I have tried following this tutorial with no luck so far: https://towardsdatascience.com/predicting-next-purchase-day-15fae5548027
Are there any other techniques to predict after how many days a customer makes his second purchase on Python?
Thanks to all in advance!


texts: Why is a lower bound necessary in proofs of VC-dimensions for various examples of hypotheses?

texts:
In the book "Foundations of Machine Learning" there are examples of proving the VC dimensions for various hypotheses, e.g., for axis-aligned rectangles, convex polygons, sine functions, hyperplanes, etc.

All proofs first derive a lower bound, and then show an upper bound.  However, why not just derive the upper bound since the definition of VC dimension only cares about the "largest" set that can be shattered by hypothesis set $\mathcal{H}$?  Since all examples ends up with a lower bound matching the upper bound, is the lower bound just helpful/useful to set a target when trying to show an upper bound?

Reference:
From page 41 of this book pdf version
https://pdfs.semanticscholar.org/e923/9469aba4bccf3e36d1c27894721e8dbefc44.pdf


texts:
I was trying to solve one problem based on Linear Regression (Predicting the sales which is a continuous variable). I have used Linear Regression for the problem but there is one suggested solution using Lasso Regression. However, the user has used Train function. I am unable to understand why train function has been used. Is it the syntax for Lasso? Similar function has been used for Ridge Regression.

my_control = trainControl(method="cv", number=5)

Grid = expand.grid(alpha = 0, lambda = seq(0.001,0.1,by = 0.0002))

lasso_linear_reg_mod3 = train(x = Train[, -c(1,2)], y = Train$Item_Outlet_Sales, method='glmnet', trControl= my_control, tuneGrid = Grid)


texts: Converting time series data to MNIST like data?

texts:
There are some time series data like this:

Time    Feature A     Feature B       Class
1          0.3           0.6            1
2          0.5           -0.2           0
3          0.8           0.3            1
...

When using RNN on MNIST, we can use 28 time steps of 28 rows as input. For these time series data, one could use 10 time steps of rows for the classification problem. (eg. Feed the rows from T1 to T10 one by one into the RNN to predict the class at T11, then the rows from T2 to T11 to predict T12 and so on.)

What is some way to convert the time series data to build an array of arrays?


texts: Help required in understanding how the error of a convolutional layer is calculated when filter and delta of next layer have differing dimensions

texts:
I am trying to implement a CNN in NumPy so as to better understand its inner workings
My architecture is as follows

10 images with 1 channel and with 28-pixel rows and columns (Dimension : (10X1X28X28))
3 convolutional filters. Each with the dimension 5X5. (Dimension: 3X1X5X5)(stride=1)
First Convolutional Output with dimension (10X3X24X24)
First Max Pooling Layer(stride=2)
First Max Pooling Output with dimension (10X3X12X12)
6 convolutional filters. Each with the dimension 3X5X5 (Dimension : 6X3X5X5) (stride=1)
Second Convolutional Output with dimension (10X6X8X8)
Second Max Pooling Layer(stride=2)
Second Max Pooling Output with dimension (10X6X4X4)
First Fully Connected Layer of Dimension (96X10)
Second Fully Connected Layer of Dimension (80X10)
Output Layer of Dimension (10X10)(Softmax Layer)

Using backpropagation, I have been able to calculate the error of the second convolutional output, which has the dimensions 10X6X8X8. I need to calculate the error of the first max-pooling output. This involves the following steps

Take the convolutional filter that connects the first max-pooling output and the second convolutional output. This has the dimensions (6X3X5X5).
Rotate the 3rd and 4th axes by 180 degrees. In code that will beresulting_weight = np.rot90(np.rot90(weight,axes(2,3),axes=(2,3))
This results in the new rotated matrix that has the dimensions (6X3X5X5)
Do full convolution between the errors of second convolutional output(10X6X8X8) and the new rotated matrix(6X3X5X5)

I am unsure of how to implement the last step due to un-matching dimensions. The full convolutional operation should result in a matrix with the dimensions (10X3X12X12). This is made possible only if the rotated matrix has the dimensions (3X6X5X5). Should we re-shape the rotated matrix to have the dimensions (3X6X5X5)? Or is my understanding of the 180-degree rotation wrong?


texts: data analysis EDA issues, indent or type

texts:
I am a newbie in DS world, right now i am working on some EDA practice, and run into an issue here.

Here is my code:

Convert some columns to numeric values

for column in cols:
    auto[column] = pd.to_numeric(auto[column]
    return auto

auto = read_auto_data()

error 1:
File "", line 4
return auto
^
SyntaxError: 'return' outside function

I tried to change return's indent:

Convert some columns to numeric values

for column in cols:
    auto[column] = pd.to_numeric(auto[column])
return auto

auto = read_auto_data()

error 2:
TypeError                                 Traceback (most recent call last)
 in ()
  1 ## Convert some columns to numeric values
  2 for column in cols:
----> 3     auto[column] = pd.to_numeric(auto[column])
  4 return auto
  5

TypeError: 'function' object is not subscriptable

Can anyone help me understand what went wrong here?

Thanks


texts: Predict items customers would buy in next order

texts:
I am working on a time series classification problem to identify what items customers would buy in their next order (customers orders different products every week).

Let's say we have a customer who orders every week but different items. Overall, this customer ordered 1k items in the past, and my task would be to identify approximately 80-120 unique items (number of items customer order each week was between 80-120) from these 1k items.

Right now, I am using the xgboost classifier, and my precision is close to 70%, and recall is 65%. I understand xgboost cannot handle seasonality by default. So I created additional features like a week of the month, week of the year, season (Spring/summer/winter/fall), etc.

I wanted to know if anyone in this community worked on a similar kind of problem. I have seen lot of peoples posting same kind of problem but not sure whether they ended up being helped/ moving it to production.


texts: Split into test and train set before or after generating document-term matrix?

texts:
I'm working on simple machine learning problems and I trying to build a classifier that can differentiate between spam and non-spam SMS. I'm confused as to whether I need to generate the document-term matrix before splitting into test and train sets or should I generate the document-term matrix after splitting into test and train?

I tried it both ways and found that the accuracy is slightly higher when the I split the data before generating the document-term matrix. But to me, this makes no sense. Shouldn't the accuracy be the same? Does the order of these operations make any difference?


texts: how can I recognize multiple faces from one image in python

texts: I wants to create an app which can recognize multiple faces from one image, I planed to built an android app through KIVI, I am not getting from where to start, it is my first minor project, any help will be appreciated.


texts: Image Classification task with unevenly sized images

texts:
Generally speaking, what's the best way to approach an image classification task with unevenly sized images (e.g. example 1 has size 300x240, example 2 has size 240x224 etc.)? Ideally I would like the dimensions to be consistent across all images before commencing CNN training.
I'm quite new to deep learning and would really appreciate any help.
Thanks


texts: Calculate implicit rating from streaming behaviour for Recommendation Engine

texts:
I have a dataset containing some user streams data for particular videos like below:
u_id|start_stream_time_dt|watch_time_ms|video_category
   1|        2021-02-01  | 3600        |  Live

My goal is to build a recommender system for watch streams.
However, I would like to find the optimal watch_stream threshold (or other approaches) that would allow me to define if a user has indeed watched a video because he/she's interested.
In other words I'd like to fill the 1s in the user_item matrix based on these information I have.
Any good approaches to suggest me? or any resources I can look up?
Thanks.


texts: Theoretically Speaking, How Do Squeeze-and-Excitation Blocks Help?

texts:
A SE block works by assigning a weight to each channel, contrary to a vanilla filter, which gives equal importance to all channels. My question is, theoretically speaking, shouldn't a regular filter be able to simply multiply the weights in each of its kernels by an amount during optimization? Of course, in practice, we might not be able to do something like that but on paper, is it possible?
Thank you in advance!


texts: How can I make a better unsupervised text classifier model? Is POS tagging part of Machine Learning and Data science?

texts:
I have got complaints data, which is not good, and often contains less than 3 words in every complaint (sometimes so short that only one word of them makes sense). The objective is to find what's trending in these complaints i.e. what people are complaining about.
I tried LDA/NMF and other topic modeling techniques but it's not performing well and mixing up different complaints in one. Also, there is no way to monitor these techniques in Production and monitor the performance (what if new types of complaints start coming in the future and the classes start mixing up? I can not fix the number of classes in LDA in production.). To add, LDA/NMF models are computationally expensive.
I built a system to identify Verbs and Nouns as keywords (using texthero - off course after extensive cleaning and lemmatization) and make classes of every complaint being received. Like 100 complaints being received are "delivery" related, another 50 are "quality" related, etc. So far, this system seems to have performed the best, and without any misclassification (obviously).
Now I have two questions:

Is POS tagging part of Data Science and Machine learning? I am in a Data science role and how can I justify this work?
Is there anything else that I can try given such a dataset?


texts:
I'm looking at working on a machine learning project for a company, where they are interested in paying in instalments at certain milestones in the project. My initial thoughts are that how to define these milestones?

It's development of a recommender system and deployment on their site.  It might take up to 12 months in worst case scenario.

I am thinking to ask for payment at these points:

before starting the project (0 months)
when I demo a prototype (low accuracy but bare bones functionality) (3 months)
when I get the prototype running on live data (higher accuracy) (6 months)
when it passes A/B testing and user evaluation (9 months)
when it's fully deployed (12 months)

What I am not sure of is, is it practical to define milestones like this? I am also wary of defining acceptance criteria like percentage accuracy since it's really hard to work out what would be acceptable.

I am also looking into this standard for ideas for milestones: https://en.wikipedia.org/wiki/Cross-industry_standard_process_for_data_mining

which would suggest Business Understanding, Data Understanding, Data Preparation, Modelling and Evaluation. However these are more fluid "phases" rather than milestones.

I'd like to find out if there's an established best way to scope and plan projects like this and how to define acceptability at each stage?

Thank you!


texts: Breaking a DataFrame based on Specific Column Values

texts:
Assume I have a following dataframe which has years of experience and salary.

I would like to create a new dataframe from the existing one, only including observations where salary is above 45000. I would like it to include both columns.


texts: Saving LSTM hidden states while training and predicting for multi-class time series classification

texts:
I am trying to use an LSTM for multi-class classification of time series data.

The training set has dimensions (390, 179), i.e. 390 objects with 179 time steps each.

There are 37 possible classes.

I would like to use a Keras model with just an LSTM and activation layer to classify input data.

I also need the hidden states for all the training data and test data passed through the model, at every step of the LSTM (not just the final state).

I know return_sequences=True is needed, but I'm having trouble getting dimensions to match.

Below is some code I've tried, but I've tried a ton of other combinations of calls from a motley of stackexchange and git issues. In all of them I get some dimension mismatch or another.

I don't know how to extract the hidden state representations from the model.

We have X_train.shape = (390, 1, 179), Y_train.shape = (390, 37) (one-shot binary vectors)/.

n_units = 8
n_sequence = 179
n_class = 37

x = Input(shape=(1, n_sequence))
y = LSTM(n_units, return_sequences=True)(x)
z = Dense(n_class, activation='softmax')(y)

model = Model(inputs=[x], outputs=[y])
model.compile(loss='categorical_crossentropy', optimizer='adam')

model.fit(X_train, Y_train, epochs=100, batch_size=128)
Y_test_predict = model.predict(X_test, batch_size=128)

This is what the above gives me:

ValueError: A target array with shape (390, 37) was passed for an output of shape (None, 1, 37) while using as loss 'categorical_crossentropy'. This loss expects targets to have the same shape as the output.


texts: Which policy gradient method is used for continuous action spaces?

texts: Which policy gradient method is used that deals with continuous action spaces?


texts: I have learned regularization for linear and logistic regression but when I implement that algorithm to my code generally my estimates not changing.I mean,it looks like ineffective.I know,it's for overfitting.So if I use it in my code every time ,could it be a problem? or is this a good thing?


texts: How do i improve my accuracy in LinearSVC? Looking for better approaches/advices

texts:
I'm struggled to get accuracy around 70 used all the tricks and tips to improve it but couldn't make it my goal is to get at least 90+ accuracy. Trained 2 folders with 4000 images 2000 images for each folder, but getting only around 69 or sometimes 70 accuracy.
Here is the code am trying, please give me suggestions to improve. thanks
import cv2
import numpy as np
import os
import glob
import mahotas as mt
from sklearn.svm import LinearSVC
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import GridSearchCV
import joblib

# function to extract haralick textures from an image
def extract_features(image):
    # calculate haralick texture features for 4 types of adjacency
    textures = mt.features.haralick(image)

    # take the mean of it and return it
    ht_mean  = textures.mean(axis=0)
    return ht_mean

def ResizeWithAspectRatio(image, width=None, height=None, inter=cv2.INTER_AREA):
    dim = None
    (h, w) = image.shape[:2]

    if width is None and height is None:
        return image
    if width is None:
        r = height / float(h)
        dim = (int(w * r), height)
    else:
        r = width / float(w)
        dim = (width, int(h * r))

    return cv2.resize(image, dim, interpolation=inter)

# load the training dataset
train_path  = "D:/ai training/aphids/blacked image"
train_names = os.listdir(train_path)

# empty list to hold feature vectors and train labels
train_features = []
train_labels   = []
# loop over the training dataset
print ("[STATUS] Started extracting haralick textures..")

for train_name in train_names:
    cur_path = train_path + "/" + train_name
    cur_label = train_name
    i = 1
    for file in glob.glob(cur_path + "/*.jpg"):

        print ("Processing Image - {} in {}".format(i, cur_label))

        # read the training image
        image = cv2.imread(file)
        #resize = ResizeWithAspectRatio(image, width=1250, height=1000) # Resize by width OR
        # convert the image to grayscale
        gray = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
        # extract haralick texture from the image
        features = extract_features(gray)

        # append the feature vector and label
        train_features.append(features)
        train_labels.append(cur_label)

        i += 1

        # otherwise create the model, train the model and save the model
#if os.path.exists("D:/ai training/aphids/joblib_model.sav"):
   # print("Loading Trained Model")
   # clf_svm = joblib.load("D:/ai training/aphids/joblib_model.sav")
#else:
        # have a look at the size of our feature vector and labels

print ("Training features: {}".format(np.array(train_features).shape))
print ("Training labels: {}".format(np.array(train_labels).shape))
print ("[STATUS] Creating the classifier..")
##Step4: Build a model
#tuned_parameters = {'kernel': ['rbf','linear'], 'gamma': [1e-3, 1e-4],
                     #'C': [10, 100, 1000, 10000]}

clf_svm = LinearSVC(class_weight="balanced", random_state=35,dual=False,tol=1e-5, C=100.0, max_iter=100000)
                       #, tuned_parameters)
#clf_svm = LinearSVC(random_state=35, dual=False,tol=1e-5, max_iter=100000)
#random_state=0, dual=False,tol=1e-5,
# fit the training data and labels
print ("[STATUS] Fitting data/label to model..")

clf_svm.fit(train_features, train_labels)

#savemodel
joblib_file = 'D:/ai training/aphids/joblib_model.sav'
joblib.dump(clf_svm, joblib_file)


texts: how to modify sparse survey dataset with empty data points?

texts:
I am working on a data set where the categorical variables have lots of empty spaces (not "NA" but ""). For example, one variable has 14587 empty spaces out of 14644 observations. There are many such variables where most of the observations are empty.In fact it is a survey dataset where the participant just chose to ignore a particular question.

I have never handled similar dataset. I am looking for advise as to how best to handle such datasets before any modeling is done. Deleting the rows or the variables with lots of empty spaces doesn't seem feasible.

Thanks a lot.


texts: Trying to figure best machine learning technique to evaluate schedules for a telescope

texts:
I work for an astronomical observatory with two geographically separated telescopes. Users submit proposals in the form of programs consisting of multiple observations, and we are trying to automize our observation scheduling system. For many years, we have had "queue coordinators" (QCs) spend several hours per day coming up with multiple schedules of observations for the night depending on weather conditions.
Right now, the science staff is having a real issue formalizing what makes for a good schedule as a mathematical equation, and I don't think it can be boiled down to a simple equation given the number of variables. My thought was that if we assume that the schedules created by the QCs are "close enough" to optimal, we can train some assisted machine learning technique (which one, I am not sure) to, being fed the data, to produce a score indicating the fitness of a proposed schedule. (We do have many years of daily historical data of schedules that we could consider close enough to optimal to use for the training and testing.)
Here are examples of some of the variables, although I don't know all of policies yet:

We divide scientific programs into bands 1, 2, 3, and 4, with band 1 programs being of the highest importance: essentially, we must complete all band 1 programs, and if the conditions are not sufficient for a band 1 program, we work on a band 2 program, etc. The higher the number, the less programs we expect to complete in that band.

A program generally has enough data to be useful for scientific purposes when it hits 80% completion; before then, it is essentially uselesss. Completeness over 80% is much less valuable than hitting the 80% mark.

We have thesis vs. non-thesis programs within each of the bands, with thesis programs being worth slightly more than non-thesis programs.

Completing a long program should be worth considerably more than completing a short program.

Observations are best scheduled near zenith (when the target is right overhead), offset by a few degrees due to the way the mirrors are arranged on the telescope.

The target of an observation obviously has to be visible at the time it is scheduled, and have a certain distance from the moon to avoid lunar interference.

We keep running into problems with proposed scoring functions by science, who seem to think that this can be modeled as a function of two inputs, i.e. band and completion rate. For example:

One proposed function motivated the scheduler to start as many band 1 programs as possible, but had much less impetus to complete them to the point of scientific use.

One proposed function made completing programs overly valuable, so the scheduler was motivated to complete as many short programs as it could, which left long programs with not enough nights to be completed.

They have just been throwing functions at each other and multiplying functions by their slopes, etc, to try to find something that comes close to capturing the observatory policy, which is not working.
I'm a combinatorial design theorist and while I have taken courses in ML years ago, I can't think of the best assisted machine learning technique to train for this kind of problem.
Does anyone have any ideas? If this is posted to the wrong stack exchange, I apologize. Please direct me to the correct one. I considered mathematics but thought it was more appropriate here.
Thanks very much for any suggestions, as we are really unsure of how to proceed.


texts:
i have a data that have some features like latitude , longitude , timestamp ,and category of an event . So the data set only has the events that actually took place . The thing is I am wondering if there is preferred models that can be used to predict the following cases :

The place ( latitiude and longitude )of next event given the current timestamp.
The high probable time that event could occur given the current place.
The probability of an event taking place given both time and space data.