texts: How to deal with a potencially multiple categorical variable

texts:
I'm build a model that has, as inputs, some categorical variables. I had already dealt with this sort of data before, and applied different techniques as creation of dummy variables and factor scoring. However, I have now a different type of problem which I can not see the obvious best answer to.

For each individual we can have multiple instances of this categorical variable $X$. When such cases happen on numerical variables I usually take the max/mean/min depending on context. I of course, one can use said context to build something similar here. However I'm curious about a general approach.

Assuming that for each object (row in our input matrix) we can have multiple entries of an categorical variable. Furthermore, assume that said variable can have many different values, and that for the context it can be relevant the combinations per row.

What would be a general approach to this variable?


texts: What is "posterior collapse" phenomenon?

texts:
I was going through this paper on Towards Text Generation with Adversarially Learned
Neural Outlines and it states why the VAEs are hard to train for text generation due to this problem. The paper states

  the model ends up
  relying solely on the auto-regressive properties of the decoder while ignoring the latent variables,
  which become uninformative.

please simplify and explain the problem in a lucid way.


texts: What is the difference between Missing at Random and Missing not at Random data?

texts:
I have been working with a dataset where the missing data seem to following a few particular patterns. I have gone through a lot websites and articles related to missing data but I haven't been able to understand the difference between MAR and MNAR.

First I would like to give the description of the dataset and the patterns of the missing data. So it would be easier for you to explain the difference between the two and also it would enable to identify the what are those patterns in my dataset.

Description of the dataset: It is a transaction history dataset of a cycle company(Imaginary company) which contains descriptions about customers like their name, DOB, gender, geographic location, Income, social status and the transaction details like the cycle brand name bought, size of the cycle(small,M,L), type of cycle like (Mountain, Road and Standard), Product manufacturing cost, List Price of the product and finally the transaction date.

Pattern in the missing data:
There are basically four different patterns of missing data that I have identified in the dataset:

Pattern 1: This is related to product details in the transaction history: Whenever there is some missing data about say the brand name then the details regarding Product line, Product class, Product size, standard cost and Product_first_sold_date are also missing but the list price alone is available but then these list prices seem unique as they cannot be found repeating in the same column. This could blamed on the company for not recording the details properly but there is actually no such company that exists and it is upto me to deal with.

Pattern 2: Whenever there is missing data about a customer say first name then all the other columns about the customer are also missing like Gender, Past_3_years_bike_related_purchases,Job_Industry_Category,
Wealth_Segment,Deceased_Indicator and Owns_Car. Only the customer ID in this case is available. Probably this could be taken as the customer didn't want to disclose the details but then again it is upto me to deal with.

Pattern 3: Whenever there is some missing data about the geographic location of the customer like say the Address then the Post code, state,country and property evaluation column data are also missing but the general customer details like name, gender, DOB are all available. Probably in this case the customer didn't want to reveal their geographic details and I have got to deal with it.

Pattern 4: Most interesting one: The gender column has three categories M, F and U. U can be taken as not disclosed. Whenever the gender is U then their age and tenure are also missing. This could be taken as those not ready to disclose their Gender are not ready to disclose their age and tenure.

Sometimes some patterns occur together.

What category of missing date like(MAR, MCAR and MNAR) do these fall into? And how do I deal with it. Any suggestion would be extremely helpful. Thanks.


texts: Does a NB wrapper consider feature subset size?

texts:
while comparing two different algorithms to feature selection I stumbled upon the follwing question:

For a given dataset with a discrete class variable we want to train a naive bayes classifier. We decide to conduct feature selection during preprocessing using naive bayes in a wrapper approach.
Does this method of feature selection consider the size of the used feature subsets?

When considering how NB classifies a given instance, the size of the feature subset being used for classification only influences the number of parts that the product of the conditional dependencies has but that does not make a difference, or does it?

It'd be great if someone could offer a solid explanation since for me it's more of a gut feeling at the moment.


texts: Pytorch doing a cross entropy loss when the predictions already have probabilities

texts:
So, normally categorical cross-entropy could be applied using a cross-entropy loss function in PyTorch or by combing a logsoftmax with the negative log likelyhood function such as follows:

m  = nn.LogSoftmax(dim=1)
loss = nn.NLLLoss()
pred   = torch.tensor([[-1,0,3,0,9,0,-7,0,5]], requires_grad=True, dtype=torch.float)
target = torch.tensor([4])
output = loss(m(pred), target)
print(output)

The thing is.  What if the data at the output is already in a state with the probabilities where the variable pred already has the probabilities.  Where the data is presented like the following:

pred = torch.tensor([[.25,0,0,0,.5,0,0,.25,0]], requires_grad=True, dtype=torch.float)

How could the cross-entropy then be completed in PyTorch?


texts:
TL;DR: Given a big image dataset (around 36 GiB of raw pixels) of unlabeled data, how can I cluster the images (based on the pixel values) without knowing the number of clusters K to begin with?

I am currently working on an unsupervised learning project to cluster images; think of it as clustering MNIST with 16x16x3 RGB pixel values, only that I have about 48 million examples that I need to cluster.
Without knowing their identities, I do know that some of the images are definitely related because they come from the same source, but - say - I also don't know an appropriate K in order to "just" run K-means on the set yet.

I was thinking of doing some manual 2D embedding using t-SNE and then clustering manually in the embedded space (a simpler task than doing it manually in 16x16x3-d), but all t-SNE implementations I could find required loading the data into memory. I also thought about first running t-SNE, then K-means on the t-SNE embedded data, but if you look at the results of t-SNE from MNIST, it's very obvious that these clusters might and probably will be distorted and skewed in nonlinear ways. So even if I were to know a K, the clusters would probably be split up. Using Mahalanobis distances for K-means might be an interesting thing, but since I don't know covariances to begin with, that appears to be a dead end as well.

Currently I'm trying if I can run PCA compression on the examples to at least gain some memory back for t-SNE, but that might or might not work ... can't say for now.

Can somebody give me a pointer in the right direction to do this (ideally, but definitely not necessary in a Python, TensorFlow or Apache Beam/Dataflow context)? I was working on porting a Streaming/Ball K-means a while ago which does have the nice property of creating new clusters "on demand", but before I start implementing that again in Python/TensorFlow/Dataflow, I was hoping that somebody could give me some ideas where to start or what to avoid.


texts:
What is the difference between sequence data and time series data?

My understanding is that sequence data is any data where the order matters and time series is a special type of sequence data ordered by the time stamps. Is this correct?

Is there a paper or book that defines it so I can cite it in a research paper?


texts: To be useful, doesn't a test set often become a second dev set?

texts:
I'm a little unclear about the expected use/value of a test set in machine learning. Here is a story that explains my confusion, assuming you're using a train/dev/test split:

You use your dev set to choose the best hyperparameters and make various tweaks, and when you're finally "done" you evaluate it on your test set. Your test performance comes back much worse than your dev performance.

So now you conclude, "My dev set must be too small, causing me to overfit my hyperparameters." So you make your dev set bigger, find new hyperparameters, and evaluate on your test set again. Now your dev and test performances are close to each other.

But note that you used your test set twice in that case. So in some sense you were fitting your hyperparameters to your test set, and it became a second development set.

To try to answer my own question: I guess you could say that the value of the test set is that without it, you would have never known you were overfitting your hyperparameters. And as long as we aren't using the test set "too much" in the above way (increasing the dev set size in a cycle again and again), it is still "mostly" unbiased. However we would have to concede that the test set is only truly unbiased if we only need it once.

Do you think this is an accurate take?

Incidentally, I'm unsure if there's anything else you can do (other than increasing dev set size) if your test performance comes back much worse. Well, I suppose you can cry ;). But are there any other options?


texts: When to use model.train() vs model.eval() in Pytoch?

texts:
I have a model that is used in a reinforcement learning algorithm for checkers, a la AlphaZero. Similar to that network, mine features batch normalization after each convolution layer. I am aware that this will cause different behavior/output when using .eval() vs .train()

However, I am unsure of when to use eval() vs train(). The model is used at two different points in the algorithm: First, the network is used to generate many games of self-play. Secondly, the network is trained using the positions of theses games, with the evaluation labels taken from the terminal value of the game (-1, 0, +1) and the 'improved policy' labels are taken to be the visit counts after the UCB-tree-search.

It seems to me that when the network is fully trained, I will use .eval(), as that should be 'what the network really thinks'. Therefore, for the games of self-play, I should also use .eval(). Ostensibly, this should result in stronger games of self-play and thus higher quality data. Finally, if I used .eval() in the self-play step I must also use it in the learning phase, otherwise if the network outputs are different the loss won't even be calculated using the actual outputs! I know that the network learns valuable information from .train(), as the batch norm layers learn about the mean/variance of the data universe.

As I type this, I am starting to suspect that I should be using .train() for both the self-play and learning phases. Still that seems wrong as the discrepancy in output between train() and eval() for a given position can be quite large.


texts:
I am new to object detection, I read the blogs for R-CNN, It is understandable.

Pass the image to the selective search, It will output different proposals.
Resize all the proposals into a fixed size.
Pass the fixed size proposals to a CNN and get the class label and Bounding box coordinates

It is the overall architecture or it is the architecture for testing the images.
But, how to do the training?
For example, I have this data set - https://github.com/Shenggan/BCCD_Dataset
This is for classify blood cells and count the number of different types of blood cells. Obviously, it is a detection problem.
Let me correct, If I am not correct...
Training, I need the input and the corresponding output/target values.
For input(X), Shall I crop each and every object(blood cells) in all images using annotation file and train only the cropped images(one object per cropped image) or train the actual images without cropping objects?
And, for output(Y), there are two outputs we are going to get from the R-CNN,

class label
Bounding box coordinates(x1, y1, x2, y2)

I am really confused here, to generate target values to train CNN. If I feed the CNN with cropped images(one object per image), then I can assign the output/target values(class labels) for every cropped image because every cropped image contains exactly one object. Plus Background.
And, How to assign bounding box coordinates(ground truth) as output for the input images to train?
In some blogs, the authors says, crop the objects using annotation file and add background images to do classification task. Then, testing images, use selective search to generate proposals with their coordinates(x1, y1, x2, y2) and feed all the proposals to the model and which class has high accuracy is the output and draw the corresponding proposal coordinates in the input image. In this technique, they did not train then bounding box regression. I followed this technique but not getting accuracy for different cases because objects are vary in size and shape.
Please, help me understand this questions, thank you
And Please, do not give down vote to this question, I hope, I explained clearly.


texts: How to model segmentation of a sequence to similar parts?

texts:
I guess LSTM is good for sequence modeling but how would you model "clustering" with it? Meaning, the input is a sequence and the output is labels with similar properties (I have labeled data). For example:

input:  1 2 1 1 2 1 2 5 6 5 4 5 1 1 2 1 1 2 1
output: 1 1 1 1 1 1 1 2 2 2 2 2 1 1 1 1 1 1 1

The problem is how to tell the model to learn differences and not the particular example, as a new example could have totally different values. I could have taught it to predict the N*(N-1)/2 pairwise differences (similar non similar) but it will not learn the dynamics.


texts: Python XGBoost predict_proba returns very high or low probabilities

texts:
I trained my data with XGBoost in python with GridSearchCV as follows:

parameters = {'nthread':[6],
              'objective':['binary:logistic'],
              'learning_rate': [0.01, 0.1],
              'max_depth': [5,8,13],
              'n_estimators': [200,500,1000,3000],
              'seed': [1337]}

xgb_model = xgb.XGBClassifier()

clf = GridSearchCV(xgb_model, parameters, n_jobs=-1,
                   cv = StratifiedKFold(shuffle=True,n_splits=5),
                   scoring='accuracy',
                   verbose=2, refit=True)

clf.fit(scaled_X_train.values, y_train)

On the test test I got 0.9 accuracy which is acceptable. However when I predict probabilities with predict_proba I saw that probabilities mostly lie between 0-0.1 and 0.9-1 ranges for 0 and 1 classes respectively.

Since I try to get scores based on the model, those dense probabilities are not so useful.

So what is the main reason of this dense probability distribution? Is this a bad thing? And how can I improve my workflow so that probabilities get wider score range?

Thanks in advance!


texts: Successful ETL Automation: Libraries, Review papers, Use Cases

texts:
I'm curious if anyone can point to some successful extract, transform, load (ETL) automation libraries, papers, or use cases for somewhat inhomogenious data?

I would be interested to see any existing libraries dealing with scalable ETL solutions.  Ideally these would be capable of ingesting 1-5 petabytes of data containing 50 billion records from 100 inhomogenious data sets in tens or hundreds of hours running on 4196 cores (256 I2.8xlarge AWS machines). I really do mean ideally, as I would be interested to hear about a system with 10% of this functionality to help reduce our team's ETL load.

Otherwise, I would be interested to see any books or review articles on the subject or high quality research papers. I have done a literature review and have only found lower quality conference proceedings with dubious claims.

I've seen a few commercial products advertised, but again, these make dubious claims without much evidence of their efficacy.

The datasets are rectangular and can take the form of fixed width files, CSV, TSV, and PSV. Number of fields range from 6 to 150 and contain mostly text based information about entities.  Cardinality is large for individual information (address), but smaller for specific details like car type (van, suv, sedan).

Mappings from abbreviated data to human readable formats is commonly needed, as is transformation of records to first-normal-form.

As is likely obvious to the cognoscenti, I am looking for techniques that move beyond deterministic methods, using some sort of semi-supervised or supervised learning model.

I know this is a tall order, but I'm curious to assess the state-of-the-art before embarking on some ETL automation tasks to help guide how far to set our sights.

Thanks for your help!


texts: How to Standardize/Normalize three sets of test data of different lengths

texts:
I have three sets of data that were sampled at the same interval (20hz). The data is from a data logger that measured vehicle speed at a specific time.

In each of the tests, the Car starts at the same starting point A, and is manually drive to the end point B. This was performed 3 times. The point at which the vehicle comes to a stop is different each time since it is manually driven but that is intended.

My intention is to gather enough data sets from this test route so if another person drives the test route, we can compare how he performed.

I would ultimately like to average the data sets from my run and call it the optimal path and add an error band, so if someone else runs the test route, I can compare how they did to it, like distance from end of route, and max speed through a turn.

Initially I thought I could plot the data vs distance and take the average that way but the problem I am running into is, log A is 400 samples, log B is 420 samples, log C is 440 samples. So they arent exactly 1:1. Any idea how to skin this cat?


texts: What are the online platforms that cater to freelance data scientists looking for work?

texts:
For example, something along the lines of the professionalism of:

https://www.kaggle.com/


texts: Extracting model equation and other data from 'glm' function in R

texts:
I've made a logistic regression to combine two independent variables in R, using pROC package and I obtain this:

 summary(fit)

Call: glm(formula = Case ~ X + Y, family = "binomial", data = data)

Deviance Residuals:
  Min       1Q     Median     3Q      Max
-1.5751  -0.8277  -0.6095   1.0701   2.3080

Coefficients:
             Estimate  Std. Error z value Pr(>|z|)
(Intercept) -0.153731   0.538511  -0.285 0.775281
X           -0.048843   0.012856  -3.799 0.000145 ***
Y            0.028364   0.009077   3.125 0.001780 **
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

Null deviance: 287.44  on 241  degrees of freedom
Residual deviance: 260.34  on 239  degrees of freedom
AIC: 266.34

Number of Fisher Scoring iterations: 4

>     fit

Call:  glm(formula = Case ~ X + Y, family = "binomial", data = data)

Coefficients:
  (Intercept)       X            Y
   -0.15373     -0.04884      0.02836

Degrees of Freedom: 241 Total (i.e. Null);  239 Residual
Null Deviance:      287.4
Residual Deviance:  260.3        AIC: 266.3

Now I need to extract some information from this data and I'm not sure about how to do it. First, I need the model equation: suppose that fit is a combined predictor called CP; could it be CP=-0.15-0.05X+0.03Y?

Then, the resulting combined predictor from the regression should present a median value, so that I can compare median from the two groups Case and Controls which I used to make the regression (in other words, my X and Y variables are N-dimensional with N = N1+N2, where N1 = Number of Controls, for which Case=0, and N2 = Number of Cases, for which Case=1).


texts:
I am trying to visualize, how does the linear regression model draw a straight line when we have multiple features and one label to predict. Like when we have 1 feature and 1 label, we can easily plot them on a 2-D graph and draw a straight line. Even if we have two features and one label, we can still plot them on a 3-D graph but I couldn't able to visualize, what will happen when we have 20 features. Will algorithm use 20-D graph or am I missing something.

I am new to machine learning. So this question can come as naive.


texts:
I have found the following definitions, but I don't really see the difference.

cross validation
Method for testing classification and prediction models. The data are randomly split into N partitions (typically N=10) and then N times a model is created from N-1 partitions and tested on the "holdout" data.

Leave one out
Every data point gets to be in a test set exactly once, and gets to be in a training set k-1 times.


texts: Deriving the update rule for mean in k-means clustering

texts:
I have a homework question that ask me to derive the update rule from the distortion metric:

It then says: "The update rule is derived by computing the gradient for each element of the k-th mean and solving for the value where the gradient is zero." Can somebody tell me what this question means? I already understand how k-means clustering works, but I need some clarification about what it means by "k-th mean" and "gradient".


texts: Why for logistic regression the error is given by [y ln(sigma(x)) + (1 − y) ln(1 − sigma(x)]

texts:
Why for logistic regression, with target values 0 or 1, it will not work to take the sum of the squares of the difference between target value and prediction, but rather:
$$
error({\bf w}) = -1/m * \sum_{i=1}^{m} [ y_i \ln (\sigma({x_i})) + (1-y_i) \ln (1 - \sigma({x_i} ) ]
$$


texts: How can I create a new column of binary values from my TfidfVectorizer sparse matrix?


texts: Create a single feature vector from the 2 edges of the vertex

texts:
Overview
Consider the 4 examples of the right angled vertex shown below. In each example, the vertex is made up of 2 line segments- A and B, which are perpendicular, and the all examples are nearly identical - except for slight displacement of the point of intersection.

In each example, I am given the coordinates of the 4 points.

Challenge
To the human eyes all the above examples look nearly identical. So, given the 4 coordinates above, how would you extract a feature vector which could be used for similarity comparison using methods like clustering or euclidean distance comparison, etc.
Thank you,
Sau


texts: How to interprete percentile information from the describe function in Pandas?

texts:
I am a bit stumped on how to interpret the percentile information you see when you call the describe function on dataframes in Pandas.
I believe I have a basic understanding of what percentile means. For example if in a test someones score 40% which ranks at the 75% percentile, this means that the score is higher than 75% of the total scores.
But I don't know how to translate this knowledge to interpret what I see from the describe function.
To illustrate, given the following:
test = pd.DataFrame([1,2,3,4,5,1,1,1,1,9])
test.describe()

This prints out something similar to this:
| count | 10.000000 |
|-------|-----------|
| mean  | 2.800000  |
| std   | 2.616189  |
| min   | 1.000000  |
| 25%   | 1.000000  |
| 50%   | 1.500000  |
| 75%   | 3.750000  |
| max   | 9.000000  |

Now I do not know how to interpret the values assigned to 25%, 50% and 75%. For example 5 out of the 10 values is set to 1, but the 50% has a value of 1.50000, clearly it is not saying 1.5 has a value of 50% because there is not even 1.5 in the data set.
Also why is 25% set to 1.000000 and 75% set to 3.750000?
I know I am interpreting this wrong hence this question! Would appreciate if someone can help understand this


texts: How to build a data analysis pipeline procedure

texts:
I have a series of scripts. Some are in R, some in Python, and others in SAS. I have built them in such a way that one code outputs a .csv file that the next code obtains and then that code outputs a .csv file, and so on...

I want to create a script that will automatically run each script in order so that the final output can be generated automatically.

What method would be best for this and can anyone point me to any examples of the procedure?


texts: Commercial Software for Interactive Machine Learning and Annotation in NLP

texts:
We are evaluating make or buy for a tool for interactive machine learning on legal text. With little labeling effort, it should enable for (ml layman) domain experts to build classifiers on a GUI.

Regarding commercial solutions, a came across prodi.gy of explosion AI.
It looks great and their spacy lib is awesome too.

What are competing commercial solutions in the (inter)active ML / semisupervised space, ideally geared towards NLP (legal would be a +)? Also looking into startups.

Thanks in advance.


texts: How to fake data based on the condition and weight

texts: I'm trying to fake data for the coffee shop. I've two features age and menu. Menu includes various type of drinks such as coffee [latte, espresso, mocca, etc], tea [milktea, lemontea], milk [freshmilk, matchamilk, etc]. What I'm trying to do is to fake menu based on the age like if the age is higher than 15, 80% of people who has the age higher than 15 will mostly order coffee randomly from the list of coffee [latte, espresso, mocca, etc], 20% will order milk and tea. Any idea on how to do this in the python?


texts: How to Minimize mean square error using Python

texts:
I want to minimise mean square error function to find best alpha value (decay rate) for my model.

Here is the description of my model:

time:              1st_month   2nd_month   3rd_month  4th_month  5th_month
Product_shipped     500            600         453      200        789

 If there is delay in products installed after shipping , we multiply by alpha

 Suppose alpha=-0.01
 We create a lower traingular matrix

 month   p1             p2           p3         p4            p5
  1    495.0249169
  2     490.0993367 588.119204
  3     485.2227668 582.2673201 439.6118267
  4     480.3947196 576.4736635 435.2376159 192.1578878
  5     475.6147123 570.7376547 430.9069293 190.2458849 750.5200159

M(1,1) is calculated as 500*(e^-alpha*month(=1))

M(2,1) is calculated as 500*(e^-alpha*month(=2))

M(2,2) is calculated as 600*(e^-alpha*month(=2))

So forth and so on.

Then Predicted Product shipment is sum across row:

Predicted_Installation
  495.0249169
  1078.218541
  1507.101914
 1684.263887
  2418.025197

We have originall Installation:

Original_Installation
   565
   1200
   1677
   1876
   2500

I want to minimise F(sum(Original_Installation-Predicted_Installation)^2) to find alpha which minimise this. How can we frame this or solve this in Python.


texts:
How can I design a Siamese Network with Triplet Loss function using the Matlab's Deep Network Designer?
I was not able to achieve this.


texts: Cross-validation for model comparison: use the same folds?

texts:
Let's say we have model M1 and model M2 that we want to compare. When we do 5-fold (say) cross validation, would the correct method to be to partition the data into F1, F2, F3, F4, and F5 and then run both models through those folds? Then would the way to assess if M2 outperforms M1 be to do a paired t-test?

I'm mostly thinking about a situation where I have the results of a cross-validation that someone else did and want to see if my model can beat their average of 80% accuracy. In that case, I would not have their exact folds or perhaps not even how many folds they used, so a paired t-test would not be possible. What would be the pitfalls of comparing to their metric or to their 5 metrics on the 5 folds when I don't know exactly how they allocated the observations into folds?


texts: Is there an R package which uses neural networks to explicitly model count data?

texts:
Ripley's nnet package, for example, allows you to model count data using a multi nomial setting but is there a package which preserves the complete information relating to a count?
For example, whereas an ordinal multinomial model preserves the ordering of the integers that make up the count, a fully developed model of count data as a GLM such as Poisson or Negative Binomial Regression includes how large the integer counts are in relation to each other.

Another phrasing might be, 'What kind of models come closest to combining the advantages of neural networks, in terms of, as an example, easily modelling non-linearity in the predictors, and count data GLMs, which are good at taking into account that the data is in fact a count?'


texts:
I have a 4 channel Numpy image that needs to be converted to PIL image in order implement torchvision transformations on image. But when I try to do this using PIL.Image.from_array(<my_numpy_image>) I get the following error.

TypeError: Cannot handle this data type

I cannot loose and get rid of any channel information right now so won't be able to discard any channels.


texts:
I'm just starting out with learning about Data Science and programming through various online course.

Was thinking a good first project would be a to create a log of what I'm learning and how long for and then work out ways that I can visualise this for use in a CV further down the road.

Was wanting to get a rough outline for how you would set a project like this up.

Was thinking of something like a simple site where I click on the learning resource I used and it asks for the amount of time I studied for and then records the data and time. Maybe more inputs for tags and percentage complete etc...

Currently learning R and Python.

So grateful for any ideas or advise you have.

Cheers


texts: How to use pymc3 to sample the mean of a Pareto random variable?

texts:
I Have a variable which is Pareto-ly distributed 'x', with unknown alpha and m.
I want to find out the distribution of its mean, so I use the following model:

with pymc3.Model() as model:

        alpha = pymc3.Uniform('alpha', 0, 0.2)
        m = pymc3.Uniform('m', 0, 2)
        x = pymc3.Pareto('x', alpha = alpha, m = m, observed = obs)
        mean_x = pymc3.Deterministic('mean_x', tt.mean(x))

        step = pymc3.Metropolis()
        trace = pymc3.sample(20000, step=step)
        burned_trace=trace[1000:]

But I only get a single value for 'mean_x', 'alpha' and 'm' every time I sample, and both 'alpha' and 'm' converge to the mean of their respective uniform distribution('alpha' converges to 0.1 and 'm' to 1. What is the correct way to sample the mean of a Pareto-ly distributed random variable?

Thanks


texts: Curve of MAPs to evaluate training progress of Mask RCNN on synthetic data

texts:
Is MAP (Mean Average Precision) a good substitute for measuring training and validation accuracy at different stages of training a machine learning model for object detection?

I am retraining a Mask RCNN (pretrained on MS-COCO) on synthetic images(with five classes) and the validation loss seems to fluctuate up and down till the 1600th epoch although the training loss goes down. I have calculated the Mean Average Precision at every 100th epoch on both training data and validation data as well and I don't know what to make of it.


texts: Bert: fine-tuning the entire pre-trained model end-to-end vs using contextual token vector

texts:
In the official github page of BERT, it mentions that:

  In certain cases, rather than fine-tuning the entire pre-trained model
  end-to-end, it can be beneficial to obtained pre-trained contextual
      embeddings, which are fixed contextual representations of each input
      token generated from the hidden layers of the pre-trained model. This
      should also mitigate most of the out-of-memory issues.

I am wondering in which cases, using only token vectors, will be more beneficial (other than out of memory issue)?


texts:
I need to design a system which can identify movie and production company names in a sentence.

The approach that comes to my mind is to train a NER Named-entity recognition system on labeled data so that it identifies the corresponding entities. But what about new entities (movie or production company name) which trained system hasn't seen, how can we tag them. Re-training the model every time with new released movies won't be feasible.

  Labeled data: Sentences with the position of words that corresponds to movie or production company name

I am a beginner in NLP any help would be appreciated


texts: How to shuffle only a fraction of a column in a Pandas dataframe?

texts:
I would like to shuffle a fraction (for example 40%) of the values of a specific column in a Pandas dataframe.

How would you do it? Is there a simple idiomatic way to do that, maybe using np.random, or sklearn.utils.shuffle?

I have searched and only found answers related to shuffling the whole column, or shuffling complete rows in the df, but none related to shuffling only a fraction of a column.

I have actually managed to do it, apparently, but I get a warning, so I figure even if in this simple example it seems to work, that is probably not the way to do it.

Here's what I've done:

import pandas as pd
import numpy as np

df = pd.DataFrame({'i':range(20),
                   'L':[chr(97+i) for i in range(20)]
                  })

df['L2'] = df['L']

df.T

    0   1   2   3   4   5   6   7   8   9   10  11  12  13  14  15  16  17  18  19
i   0   1   2   3   4   5   6   7   8   9   10  11  12  13  14  15  16  17  18  19
L   a   b   c   d   e   f   g   h   i   j   k   l   m   n   o   p   q   r   s   t
L2  a   b   c   d   e   f   g   h   i   j   k   l   m   n   o   p   q   r   s   t

For now, L2 is simply a copy of column L. I keep L as the original, and I want to shuffle L2, so I can visually compare both. The i column is simply a dummy column. It's there to show that I want to keep all my columns intact, except for a fraction of L2 that I want to shuffle.

n_rows=len(df)
n_shuffle=int(n_rows*0.4)
n_rows, n_shuffle

(20, 8)

pick_rows=np.random.permutation(list(range(n_rows)))[0:n_shuffle]
pick_rows

array([ 3,  0, 11, 16, 14,  4,  8, 12])

shuffled_values=np.random.permutation(df['L2'][pick_rows])
shuffled_values

array(['l', 'e', 'd', 'q', 'o', 'i', 'm', 'a'], dtype=object)

df['L2'][pick_rows]=shuffled_values

I get this warning:

C:\Users\adumont\.conda\envs\fastai-cpu\lib\site-packages\ipykernel_launcher.py:1: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  """Entry point for launching an IPython kernel.

df.T

I get the following, which is what I expected (40% of the values of L2 are now shuffled):

    0   1   2   3   4   5   6   7   8   9   10  11  12  13  14  15  16  17  18  19
i   0   1   2   3   4   5   6   7   8   9   10  11  12  13  14  15  16  17  18  19
L   a   b   c   d   e   f   g   h   i   j   k   l   m   n   o   p   q   r   s   t
L2  e   b   c   l   i   f   g   h   m   j   k   d   a   n   o   p   q   r   s   t

You can see the notebook here (it's rendered better on nbviewer than here): https://nbviewer.jupyter.org/gist/adumont/bc2bac1b6cf7ba547e7ba6a19c01adb6

Thanks in advance.


texts:
My colleague mentioned that a data science project is using SGD classifier.

So I started reading about GD/SGD and came across a nice article about Text classification using SVM and GD.

In the end of the article the author mentions:

  We begin with a bunch of documents to train on. We then represent each
  documents as numerical vectors, and you can choose to split them into
  n-grams and weigh these n-grams with TF-IDF. Now that we have a
  numerical representation, we can give the program training data and
  make it guess each time. Based on how wrong the program was, the
  program adjusts to its mistakes to make better prediction the next
  time using Gradient Descent. It keeps track of its learning using SVMs
  and make predictions with the SVM model by “plotting” the new
  datapoint in the graph and seeing if it is on the right or left by the
  decision boundary.

The article does not mention about SGD, but from internet research I understand that SGD is a variation of GD where-in the parameter is tweaked immediately at each iteration, rather than at the end of the entire training set.

So to me it appears that SVM is the actual classifier and GD (or SGD) is the function that tells the classifier how correctly it has done the prediction based on which the SVM classifier improves itself.

Coming back to my original question: Is my understanding right that SGD is not the classifer?


texts:
Aside from using validation data to tune the hyperparameters is there any other benefit to including validation data to the model?

All I ever read about is it being used to tune hyperparameters and check for overfitting. Is the checking for overfitting separate from tuning the hyperparameters?

Training: Tune the parameters (weights and biases)
Validation: Tune hyperparameters
Test: Evaluate the model

So, if we are NOT tuning the hyperparameters, the validation set is pointless?


texts:
For example, if I run a single round (nrounds=1), how does XGBoost go about making predictions?  I thought it would simply return a linear regression model, but I quickly shot that theory down by working out an example.

Please note that I've already seen this and this and most of Google's top search results.


texts: Why don't convolutional computer vision networks use horizontally - symmetric filters?

texts:
If, for example, I have a neural network for classifying dog breeds, and I feed it an image of some dog, inherently it shouldn't matter whether I feed it the original image or the image, mirrored horizontally.
I'd like to implement this symmetry in the network, and by my understanding of CNNs that means i'll generally need all the filters to be horizontally symmetric, and then the network would both be more robust and would take almost 40% less time to train assuming filter size = 5.

But,
by my research modern networks don't use filters like that. Well... Why? what disadvantages does this architecture have that I can't find a single mention of such idea?


texts: Regression performance varies hugely on shuffling training and testing data

texts:
I'm working on a regression problem to predict a variable y based on an input vector X with about 10 columns. To split the data for training and testing, I use the test_train_split method:
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle = True)

Here, it can be seen that X and y are shuffled entirely along with their indices.
For the purpose of this example, I will consider only one algorithm, RandomForestRegressor:
regr1 = RandomForestRegressor(n_estimators = 200)
regr1.fit(X_train, y_train)
y_pred = regr1.predict(X_test)

When I train the regressor with X_train and y_train from the above method, I get certain results which I consider good. For my application, I would need the model to predict for cases with sequential data (as they would be if not shuffled).
Therefore, I tried the following method to shuffle the training data alone and keep the testing data as is.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, shuffle = False)
train_data = pd.concat([X_train, y_train], axis = 1)
train_data = train_data.sample(frac=1)
X_train = train_data.iloc[:,0:-1]
y_train = train_data.iloc[:,-1]

In the above method, I first split the data into test/train indices and then shuffle the training data alone and keep the testing data with their original indices. When I train the regressor with the exact same parameters as before and test with X_test, I get significantly poorer results.
I have also tried with both train and test data without shuffling and got bad results as well. I want to be able to train the model so that it can predict for unknown values coming in a sequential order (as it would be in real time).
I'm not able to understand why the shuffling of the test data alone affects the performance, as the model should merely be predicting based on the trained parameters which entirely depend on the training data.


texts: how can autoencoder reduce dimensionality?

texts: I can't understand how is dimensionality reduction achieved  in autoencoder since it learns to compress data from the input layer into a short code, and then uncompress that code into the original data I can' t see where is the reduction: the imput and the putput data have the same dimensionality?


texts: What's the difference between GridSearchCrossValidation score and score on testset?

texts:
I'm doing classification using python. I'm using the class GridSearchCV, this class has the attribute best_score_ defined as "Mean cross-validated score of the best_estimator".
With this class i can also compute the score over the test set using score.
Now, I understand the theoretical difference between the two values(one is computed in the cross validation, the other is computed on the test set), but how should I interpret them? For example, if in case 1 I get these values (respectively as best_score_ and as score on the test set) 0.9236840458731027 0.8483477781024932, and in case 2 these other values 0.8923046854943018 0.8733431353820776, which one should I prefer and why? Why can the difference between the two values vary so much?


texts: Do you apply confidence threshold before calculating mAP for YOLOv2?

texts:
I've been using YOLO v2 for object detection and have been trying to use mAP for validating the model.

For training I use a confidence threshold of 0.25, however I am unsure as to whether to apply this threshold to filter out predictions BEFORE calculating mAP in validation? or whether the threshold should be ignored and all proposals should be evaluated?


texts:
I have a collected data from 50 unique blocks, and then merged data from 49 blocks into one data set, and saved the data from 1 block for testing purpose.
I then split the merged data set from 49 blocks using train_test_split(sklearn). Then used training data to train a random forest regressor using cross validation and get a good model score(R^2 score from sklearn random forest regressor model) on train (0.99) and test set(0.94). But when I use the trained model on the reserved data from the 1 block, the performance is very bad (-1.0).
If I merge the data from all 50 blocks, then use train test split and keep 60% data as training set, 20% as test set, and 20% (reserved set), I get good scores from all three sets. Training set score(0.98), test set(0.93) and the reserved set(0.96).
Any intuition regarding what could be causing this? And any suggestions on how to improve the model score for the 1 block of unseen data?


texts: What is the novelty in AlphaGo, Google Deepmind's Go playing system?

texts:
Recently researchers at Google DeepMind published a paper, where they described a Go playing system that beat the best current computer programs and the human European champion.

I had a quick look at that paper, and it seems it is using many interesting ideas from previous papers. What did they do differently that allowed them to achieve this spectacular improvement?


texts: I have a data set of transactions and want to build a fraud detection model (classifier). Only 3 variables are given that could be used as input features. The number of transactions during past 3, 6 and 12 months. These three features are highly correlated. I want to use the information content in all three features as much as possible. What is the best way to handle their great amount of correlation.


texts: help understanding deep Q learning algorithm from deep mind paper

texts:
I'm studying this paper https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf, and I have a question about proposed algorithm(it appears on page 7 of the paper):

The problem I see is that author's function Q (and Q^) take a state as input and outputs values, corresponding to quality of all actions agent can take. However, in the algorithm the information is structured in the form (state, performed action, reward, new state), and this is used to compute target value yj. As I understand, yj must have [number of actions] dimensions, because that is what Q returns.

My question is how do I compute yj if I know reward only for one action(which was performed by agent)?

I found a tutorial https://keon.github.io/deep-q-learning/, where target values yj are initialized as Q(state) and yj[performed action] is initialized as in algorithm.
I also implemented tutorial for snake game colab, agent learns slow, probably because of incorrect target values


texts:
Here is a line of code

str(x).split(' ')[-1][1:]

The split function within its parenthesis has ' '.
Is it necessary, or we can also leave the split function with no arguments like

str(x).split()[-1][1:]

I think both works in the same way.


texts: Siamese network return same loss value while training

texts:
I have ~9k training samples, the same pair vectors are labeled as 0 and not same pair samples are labeled as 1.

I trained it for 100 epochs for the first 3 epochs the loss value is fluctuating, then it returns the same loss value until the end.

Base network architecture:

def create_base_network():

    a = 'tanh'
    model = Sequential()
    model.add(Dense(self.INPUT_DIM, input_shape=(self.INPUT_DIM, ), activation=a))
    model.add(Dense(600, activation=a))
    model.add(Dense(600, activation=a))
    model.add(Dense(900, activation=a))
    model.add(Dense(1000, activation=a))
    model.add(Dense(5000, activation=a))
    model.add(Dense(1000, activation=a))
    model.add(Dense(900, activation=a))
    model.add(Dense(600, activation=a))
    model.add(Dense(600, activation=a))
    model.add(Dense(self.INPUT_DIM, activation=a))
    return model

Euclidean distance:

K.sqrt(K.maximum(K.sum(K.square(x - y), axis=0, keepdims=True), K.epsilon()))

Initilization model:

    base_network = self._create_base_network()
    input_a = Input(shape=(self.INPUT_DIM,))
    input_b = Input(shape=(self.INPUT_DIM,))

    processed_a = base_network(input_a)
    processed_b = base_network(input_b)

    distance = Lambda(self._euclidean_distance, output_shape=self._dist_output_shape)([processed_a, processed_b])
    model = Model(inputs=[input_a, input_b], outputs=distance)

Compilation of model:

model.compile(loss='mse',
          metrics=['mse'],
          optimizer=optimizers.Adam()
          )

Training the model:

model.fit([self.train_d['vec1'], self.train_d['vec2']], self.train_d['label'], batch_size=128, epochs=self.args.epochs, shuffle=True)

Log statements:

Epoch 1/100
9544/9544 [==============================] - 6s 594us/step - loss: 0.4556 - mean_squared_error: 0.4556
Epoch 2/100
9544/9544 [==============================] - 4s 470us/step - loss: 1.0693 - mean_squared_error: 1.0693
Epoch 3/100
9544/9544 [==============================] - 4s 464us/step - loss: 0.7328 - mean_squared_error: 0.7328
Epoch 4/100
9544/9544 [==============================] - 4s 465us/step - loss: 0.7328 - mean_squared_error: 0.7328
Epoch 5/100
9544/9544 [==============================] - 4s 461us/step - loss: 0.7328 - mean_squared_error: 0.7328
Epoch 99/100
9544/9544 [==============================] - 4s 470us/step - loss: 0.7328 - mean_squared_error: 0.7328
Epoch 100/100
9544/9544 [==============================] - 4s 470us/step - loss: 0.7328 - mean_squared_error: 0.7328

Can someone help me to find out what's the issue here?

Thanks in Advance.


texts: How to make a gaussian distribution in python considering mean. variance. skewness and kurtosis?

texts:
np.random.normal(mean,sigma,size) allows to create a gaussian distribution based only on mean and variance. I want to create a distribution based on function_name(mean,sigma,skew,kurtosis,size).
I tried scipy.stats.gengamma but I don't understand how to use it. It takes 2 parameters - a,c and creates a distribution. But it is difficult to interpret for what values of a & c, the function will give a particular value of skewness and kurtosis.
Can anyone explain how to use gengamma or any other way to create such a distribution in python, even from scratch by writing mathematical equations?
Edit: By Gaussian, I mean that I want the distribution to be normal with some skewness or kurtosis as well. It need not be a standard normal distribution.


texts: Supervised Learning with Necessarily Missing Data

texts:
Many discussions of missing data in supervised (and unsupervised) learning deal with various methods of imputation, like mean values or EM. But in some cases the data will be missing as a necessary consequence of the data generation process.

For instance, let's say I'm trying to predict students' grades, and one of the inputs I want to analyze is the average grades of the student's siblings. If a particular student is an only child, then that value will be missing, not because we failed to collect the data, but because logically there is no data to collect. This is distinct from cases where the student has siblings, but we can't find their grades.

Other examples abound: say we're in college admissions and we want to include students' AP exam results, but not all students took AP exams. Or we're looking at social network data, but not all subjects have facebook and/or twitter accounts.

These data are missing, but they're certainly not missing at random. And many algorithms, such as all supervised learning packages in scikit-learn, simply demand that there be no missing values at all in the data set.

How have people dealt with this in the past, and what off-the-shelf solutions are there? For instance, I believe the gradient boosting algorithm in R uses trees with three possible branches: left, right, and missing. Any other alternatives out there?


texts: Train a deep reinforcement learning model using two computers

texts:
I would like to know if there is a way to train a deep rl model using two different computers. The first one would execute the game and send requisitions to the second computer which would store and train the model itself.

Obs: The computers aren't in the same LAN.

Thanks!!


texts:
I have recently been trying to train a randomForest model on a binary outcome with a very uneven class split.

282 control ~82%
63 case ~18%

There are a total of 147 predictors that I'm testing for this model.

I have been running Monte Carlo cross validations to determine the best mtry parameter however I have ran into an odd issue.

Based on these plots it looks like my classifier is no better than random (when you consider the class imbalance). The part that confuses me is that for some reason Kappa seems to increase as Accuracy decreases.. which doesn't make a ton of sense to me. Furthermore the graphs seem to suggest that the best mtry in terms of the kappa statistic is equal to the number of predictors, which I am having trouble interpreting the significance of. Does this mean my model is not useful at all for feature importance selection?


texts:
There's a way to use information of one column as restriction for where in the same query? Picture that:
I have my user_table like so:

user_id
name
age
gender
lvl

1
Mike
36
male
17

2
ASh
34
female
19

And other table called services like so:

service_id
name
restrictions

1
service 1
'gender' = 'male', 'age' > 18

2
service 2
'gender' = 'female'

There is a way to do in one query a request that i just pass the user_id and its return all services that user is eligible based on restrictions of service table, i tried to do that using JS but would like to know if there is a way to do this in SQL because i believe to do more performatic.


texts: Detecting smoke and bluring/filtering using CNN in python

texts: I need some help to make an CNN via python to detection of fire-smoke and make the fire-smoke blur. What should I do where can I start some help please?


texts: Is there an algorithm or NN to match two documents, basically not closely similar?

texts:
Is there an algorithm or NN to match two documents? One is a claim description (e.g. a CV or product offer) and another is a requirements description (e.g. vacancy description or RFP). They are not similar, so basically it's not a docs similarity per se.

What's it better embedding to use on document corps (Doc2vec, Word2vec or just TF-IDF? etc) and what kind of further NN architecture would work to basically find a matching scores vector/matrix as output on how do input claim docs match to requirement docs?
Or is there exists just any text analitics algorithm or something?

Thanks in advance for help.


texts: How to cluster texts by most relevant words

texts:
I have a huge amount of documents and every document has its own portrait, where a portrait has this structure (document_id, word, weight). TFIDF, basically.
I want to cluster these documents into different clusters, say, 10.
I'm trying to implement the K-Means algorithm with sklearn, but I have almost zero experience with data science whatsoever. All tutorials that I found get texts as input from Wikipedia or somewhere else, but I don't have access to the texts themselves. I have only their portraits. Hope that makes sense.
Is this something that can be achievable with sklearn and if so, can you guide me where to dig or what to look at


texts: Getting different graph after converting tflearn model to keras

texts:
I have the following code written using tflearn:

import tflearn
from tflearn.layers.core import fully_connected, input_data
from tflearn.layers.estimator import regression

def model():
    network = input_data(shape=[None, 5, 1], name='input')
    print('network:', network)
    # Outputs: network: Tensor("input/X:0", shape=(?, 5, 1), dtype=float32)
    network = fully_connected(network, 25, activation='relu')
    print('fully_network_1:', network)
    # Outputs: fully_network_1: Tensor("FullyConnected/Relu:0", shape=(?, 25), dtype=float32)
    network = fully_connected(network, 1, activation='linear')
    print('fully_network_2:', network)
    # Outputps: fully_network_2: Tensor("FullyConnected_1/BiasAdd:0", shape=(?, 1), dtype=float32)
    network = regression(network, optimizer='adam', learning_rate=self.lr,
                         loss='mean_square', name='target')
    print('reg:', network)
    # Outputs: reg: Tensor("FullyConnected_1/BiasAdd:0", shape=(?, 1), dtype=float32)
    model = tflearn.DNN(network, tensorboard_dir='log', tensorboard_verbose=3)
    return model

The created model is trained using the following code:

def train_model(train_x, train_y, model):
    train_x = train_x.reshape(-1, 5, 1)
    train_y = train_y.reshape(-1, 1)
    print(train_x.shape)
    # Outputs: (599068, 5, 1)
    print(train_y.shape)
    # Outputs: (599068, 1)
    model.fit(train_x, train_y, n_epoch=3, shuffle=True, run_id='model.tflearn')
    model.save('model.tflearn')

When I display the graph using Tensorboard, I get the following graph:

I wanted to migrate to Tensorflow 2.2.0 (from Tensorflow 1.15.0), so I decided to rewrite the previous code using Tensorflow2, here is my try:

import tensorflow as tf

def build_model():
    model = tf.keras.Sequential()
    model.add(tf.keras.layers.Dense(units=25, activation='relu',
                                    input_shape=(5, 1)))
    model.add(tf.keras.layers.Dense(units=1, activation='linear'))
    opt = tf.keras.optimizers.Adam(learning_rate=0.01, beta_1=0.9,
                                   beta_2=0.999, epsilon=1e-07,
                                   amsgrad=False, name='Adam')
    model.compile(optimizer=opt, loss='mse')
    print(model.summary())
    return model

The model summary prints:

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
dense (Dense)                (None, 5, 25)             50
_________________________________________________________________
dense_1 (Dense)              (None, 5, 1)              26
=================================================================
Total params: 76
Trainable params: 76
Non-trainable params: 0
_________________________________________________________________
None

The created model is trained using the following code:

import datetime
import tensorflow as tf

def train(model, train_x, train_y):
    train_x = train_x.reshape(-1, 5, 1)
    train_y = train_y.reshape(-1, 1)
    # train_x.shape = (599068, 5, 1)
    # train_y.shape = (599068, 1)
    log_dir = "logs/fit/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
    tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)

    model.fit(train_x, train_y, epochs=3, verbose=1, callbacks=[tensorboard_callback])
    model.save_weights('model.h5')

When I display the model's graph I got this:

This is totally different from the graph I got with tflearn.
Did I miss something when building the model with Tensorflow?

PS: I got different losses when I train both models, the first model's loss is 0.06, while the model built with Tensorflow2 has 0.25 loss, which is far from the first one!


texts: Efficient database model for storing data indexed by n-grams

texts:
I'm working on an application which requires creating a very large database of n-grams that exist in a large text corpus.

I need three efficient operation types: Lookup and insertion indexed by the n-gram itself, and querying for all n-grams that contain a sub-n-gram.

This sounds to me like the database should be a gigantic document tree, and document databases, e.g. Mongo, should be able to do the job well, but I've never used those at scale.

Knowing the Stack Exchange question format, I'd like to clarify that I'm not asking for suggestions on specific technologies, but rather a type of database that I should be looking for to implement something like this at scale.


texts: Classification Algorithms For Noisy Problems

texts:
Dear fellow data scientists

I work in finance and have decent experience with neuronal networks. Right now I am facing a challenge where I need to classify a set of signals.

The data set looks roughly as follows (signal can be thought of as a vector which might also contain floats instead of being binary):

Signal - Target
001110   0
001110   1
001100   1
011100   0
...      ...

The problem is that the data is really noisy. That means that most of the signals do not really have a lot of predictive power to forecast the target. That makes using a neuronal network (or other classical classification algorithm) rather impractical since we will be fitting a lot of noise.
At the same time, simply looking at correlations between signals and targets does not help, since the signals themselves are noisy. So e.g. [0,0,0,1,1,1] and [0,0,0,1,1,0]  are not the same, but similar and both signals might be a very good predictor for, let's say, the target 1.

My goal is to have an algorithm that tries to find the 10 good signal "types" (in a fuzzy way) in the haystack of 1000s of signals. I am looking for suggestions as to what that algorithm could be.

One idea would be to first cluster the signals and then use the clusters as input, but maybe there are better suited algorithms exactly for such tasks.

Hope my question is clear. :)


texts: "Stationarity of statistics" and "locality of pixel dependencies"

texts:
I'm reading the ImageNet Classification with Deep Convolutional Neural Networks paper by Krizhevsky et al, and came across these lines in the Intro paragraph:

  Their (convolutional neural networks') capacity can be controlled by varying their depth and breadth, and they also make strong and mostly correct assumptions about the nature of images (namely, stationarity of statistics and locality of pixel dependencies). Thus, compared to standard feedforward neural networks with similarly-sized layers, CNNs have much fewer connections and parameters and so they are easier to train, while their theoretically-best performance is likely to be only slightly worse.

What's meant by "stationarity of statistics" and "locality of pixel dependencies"? Also, what's the basis of saying that CNN's theoretically best performance is only slightly worse than that of feedforward NN?


texts: KerasRegressor serialize/save a model as a .h5df

texts:
I am been using a script from machinelearningmastery on Keras regression and I would like to save model as a .h5 file.

Machinelearningmastery also has another tutorial for saving models/pickles but the scripts are written in a model.fit() method in Keras… But the script I am using I am defining the model thru calling a function.

Can someone give me a tip on how I can save this model as a .h5df?

import numpy as np
import pandas as pd
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
import matplotlib.pyplot as plt
import math
from sklearn.preprocessing import MinMaxScaler

# load dataset
dataset = pd.read_csv("joinedRuntime2.csv", index_col='Date', parse_dates=True)

print(dataset.shape)
print(dataset.dtypes)
print(dataset.columns)

# shuffle dataset
df = dataset.sample(frac=1.0)

# split into input (X) and output (Y) variables
X = np.array(df.drop(['kWh'],1))
Y = np.array(df['kWh'])

def wider_model():
    # create model
    model = Sequential()
    model.add(Dense(20, input_dim=7, kernel_initializer='normal', activation='relu'))
    #model.add(Dense(28, kernel_initializer='normal', activation='relu'))
    #model.add(Dense(21, kernel_initializer='normal', activation='relu'))
    #model.add(Dense(14, kernel_initializer='normal', activation='relu'))
    model.add(Dense(10, kernel_initializer='normal', activation='relu'))
    model.add(Dense(1, kernel_initializer='normal'))
    # Compile model
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model

# fix random seed for reproducibility
seed = 7
np.random.seed(seed)
estimators = []
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasRegressor(build_fn=wider_model, epochs=200, batch_size=5, verbose=0)))
pipeline = Pipeline(estimators)
kfold = KFold(n_splits=10, random_state=seed)
results = cross_val_score(pipeline, X, Y, cv=kfold)
print("Wider: %.2f (%.2f) MSE" % (results.mean(), results.std()))
print("RMSE", math.sqrt(results.std()))


texts: 0.1 accuracy on MNIST fashion dataset following official Tensorflow/Keras tutorial

texts:
My goal is to classify products pictures into categories such as dress, sandals, etc.

I am using the MNIST fashion dataset, following this official tutorial word-per-word: https://www.tensorflow.org/tutorials/keras/basic_classification so my code is 100% identical to what can be read there:

# TensorFlow and tf.keras
import tensorflow as tf
from tensorflow import keras

# Helper libraries
import numpy as np
import matplotlib.pyplot as plt

print(tf.__version__)

fashion_mnist = keras.datasets.fashion_mnist

(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()

class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']
model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(128, activation=tf.nn.relu),
    keras.layers.Dense(10, activation=tf.nn.softmax)
])

model.compile(optimizer=tf.train.AdamOptimizer(),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(train_images, train_labels, epochs=5)

test_loss, test_acc = model.evaluate(test_images, test_labels)

print('Test accuracy:', test_acc)

Problem: The resulting accuracy is always around 0.1, much lower than the tutorial's example output of 0.876.

I am obviously doing something wrong, but I can't figure out what. How to improve accuracy to something reasonable?

My output:

$ python classify-products.py
1.10.1
Epoch 1/5
2018-09-18 13:33:46.971437: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
60000/60000 [==============================] - 3s 47us/step - loss: 13.0161 - acc: 0.1924
Epoch 2/5
60000/60000 [==============================] - 3s 46us/step - loss: 12.8998 - acc: 0.1997
Epoch 3/5
60000/60000 [==============================] - 3s 46us/step - loss: 13.3386 - acc: 0.1724
Epoch 4/5
60000/60000 [==============================] - 3s 47us/step - loss: 12.9031 - acc: 0.1995
Epoch 5/5
60000/60000 [==============================] - 3s 47us/step - loss: 13.6666 - acc: 0.1521
10000/10000 [==============================] - 0s 26us/step
('Test accuracy:', 0.1005)

Switching to 20 epochs does not improve accuracy.


texts: Binary classification algorithm where the input variables are arrays

texts:
For a project, I'm trying to predict leaks in a network. The network consists of nodes connected by links. What I have are several 'scenarios' where each scenario has a leak present at a different node (it is also possible that there isn't a leak).
I have pressure data for each node, as well as the flow data for all the links. I also have the targets for all the scenarios (0/1, depending on if there's a leak or not). The data can be represented as below:

Scenario
Node 1
Node 2
Output

1
[69.887, 70.383, ...]
[71.875, 72.421, ...]
0

2
[60.358, 61.530, ...]
[76.840, 77.389, ...]
1

In case there is a leak present, during the time the leak exists, the pressure takes a massive dip and the flow shoots up. This is applicable to most nodes and links in the system, depending on where they're located.
Pressure when leak exists:

Flow when leak exists:

My initial idea was to represent all the data at a particular node as a single value using either standard deviation or mean, but that would result in the loss of the data's trends.
I wanted to see how I could possibly train a model while either keeping the data as is or transforming it in some way.
I was considering using a Random Forest classifier for this, but I'm open to suggestions.


texts: What is the meaning of the sparsity parameter

texts:
Sparse methods such as LASSO contains a parameter $\lambda$ which is associated with the minimization of the $l_1$ norm. Higher the value of $\lambda$ ($>0$) it means that more coefficients will be shrunk to zero. What is unclear to me is that how does this method decide which coefficients to shrink to zero?

If $\lambda =  0.5$ then does it mean that those coefficients whose values are less than or equal to 0.5 will become zero? So in other words, whatever be the value of $\lambda$, the coefficients whose values fall within $\lambda$ will be turned off/become zero? OR is there some other meaning to the value of $\lambda$?
Can $\lambda$ be negative?


texts: Choosing the right parameters to train a Tf-Idf vectoriser

texts:
I'm very new to the DS world, so please bear with my ignorance.

I'm trying to analyse user comments in Spanish. I have a somewhat small dataset (in the 100k's -- is that small?), and when I run the algorithm in a, let's say, naïve way (scikit-learn's default options +remove accents and no vocabulary / stop words) I get very high values for very common and low value words (such as spanish equivalents of "to", "at", etc.).

What would be the most effective way to train the vectoriser on a 100k long corpus of ~200 char long docs?

I was thinking of using a larger corpus of Spanish text or looking for a stop word removing implementation for Spanish, but would love to have some expert advise before jumping into it.

Thanks!


texts: What is the best way to predict multiple outcome from a single entity?

texts:
Let's say i have three model: Facial recognition, Face landmark detection, Emotion recognition.
Now if i want to predict those three feature from a single image. What should be my approach?

Should i combined those three model? or
Run three model in three different thread?


texts: Using an ontology to recognize named entities in free text

texts:
I'm trying to solve a fairly basic problem in NPL efficiently. What tool or software package would you use to identify the words, or group of words that are part of an given ontology within a free text.

Let's imagine the inputs are the following dummy ontology:

And this publication's abstract:

  This study evaluates the addition of metformin to standard of care in locally advanced and metastatic prostate cancer, half the patients will receive metformin in combination with standard treatment, and the other half will receive the standard of care only. Advanced-stage PCa is usually treated with androgen-deprivation therapy (ADT). Most patients with metastatic disease who were managed with ADT eventually progress to castration-resistant prostate cancer (CRPC) and die of the disease.

When run together through the tool via an API, or any such other call, then the outputs would look something like (in the format class:name text_loc_start:text_loc_end):

- ID005:ID011 34:56 (Drug:Metformin "metformin")
- ID004:ID008 117:132 (Condition:Prostate cancer "prostate cancer")
- ID005:ID011 444:523 (Drug:Metformin "metformin")
- ID004:ID008 981:995 (Condition:Prostate cancer "prostate cancer")

I am looking for a solution that would have an unrestrictive license and can be used commercially for free. Thanks in advance.


texts: Naive bayes, all of the elements in predict_proba output matrix are less than 0.5

texts:
I've created a MultinomialNB classifier model by which I'm trying to label some test texts:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn import preprocessing
from sklearn.naive_bayes import MultinomialNB

tfv = TfidfVectorizer(strip_accents='unicode', analyzer='word',token_pattern=r'\w{1,}',
                      use_idf=1,smooth_idf=1,sublinear_tf=1)

# df['text'] is a long string text of words
tfv.fit(df['text'])

lbl_enc = preprocessing.LabelEncoder()

# df['which_subject'] is one of the following 7 subjects: ['Educational', 'Political', 'Sports', 'Tech', 'Social', 'Religions', 'Economics']
y = lbl_enc.fit_transform(df['which_subject'])

xtrain_tfv = tfv.transform(df['text'])

# xtest_tfv has 7 samples
xtest_tfv = tfv.transform(test_df['text'])

clf = MultinomialNB()
clf.fit(xtrain_tfv, y)

y_test_preds = clf.predict_proba(xtest_tfv)

Now y_test_preds is as follows:

0.255328    0.118111    0.129958    0.123368    0.119301    0.131098    0.122836
0.122814    0.265444    0.117637    0.13531     0.116697    0.122812    0.119286
0.131485    0.114459    0.258224    0.122414    0.118132    0.134005    0.12128
0.125075    0.131948    0.122668    0.258655    0.116518    0.119995    0.12514
0.124356    0.116987    0.121706    0.119796    0.266172    0.127231    0.123751
0.132295    0.1192      0.13366     0.119445    0.123186    0.257318    0.114895
0.126779    0.118406    0.123723    0.127393    0.122539    0.117509    0.263652

As you see, all of the elements are less than 0.5. Does this table show anything? Can I conclude that the classifier is not able to label test text?


texts:
I am new in using python for data science.
What is the difference between selecting a a column with: df['name'].values and df.iloc[:,1].values and df.iloc[:,1:2].values they return differnt types of numpy vectors. why?


texts: How to make fair comparison of multi-task RL models if I have unlimited test data?

texts:
The data comes from a simulator hence I have the possibility to generate unlimited data. The reward is 0 (no success) and success(1) if episode is successful. Now, the question is what metric to use for evaluation? For example, if I choose the average success rate as metric and hypothetically someone trains the RL model on one episode and the reward is 1 then the average will be 1. So I have to choose min. number of episodes. Should it be 10, 1000, 10000 and how do I calculate it from the data? Each episode is separate task so each episode might be different length than the previous one and might have different complexity.
Thank you.


texts: Can PCA be applied to reduce dimensionality of only a subset of features?

texts: Lets say I have a feature set of f0 to f1000. I am thinking of applying PCA on f500 to f1000 reducing their dimensionality. Can I combine this reduced set with the features f0 to f499 as the feature space for training a learning algorithm?


texts: Textbook definition of Data augmentation

texts:
I'm trying to find a textbook definition of Data augmentation. We all know what it is, but I'm having a hard time finding a reliable source which I can cite for my paper:

The Wikipedia article on Data augmentation misquotes an article by Shorten & Khoshgoftaar which doesn't actually present a definition.
There are many definitions in blog posts, but these are not reliable sources.

It doesn't help that there is a Markov chain Monte-Carlo algorithm with the same name cluttering up search results.
Any clues?


texts:
I want to change the values of the class labels from nominal into numeric.
e.g if the values of a class are {iris-setosa,iris-virginica,iris-versicolor} i want to make them {0,1,2} so the instances will have as a value at the class label the form (0,0,1,2,0,1,1,2,0).
Any idea?


texts:
I am performing 10-folds cross-validation to evaluate the performances of a series of models (variable selection + regression) with R. I created manually the folds with this code.

At the moment I'm performing first variable selection, then hyperparameters tuning through cv, and finally testing the performance with RMSE and MAE for all the models, but I have a doubt.

Is it correct to "use" the same fold for all the models? Or should I do a separate cv for each model?


texts:
I want to build a model to predict if a customer will convert from a free product to its premium version. In addition, I'm also interested in knowing when she will convert.
This is different from the usual conversion prediction when we want to know if there's a conversion, independently of when it happens.
I've organized my conversion prediction in monthly buckets in the following way. User A has a probability of conversion:

0 in month 1
0.1 in month 2
0.1 in month 3
0.1 in month 4
0 in months 5-12
0.7 to not convert in the first 12 months

The targets that we train on should be something like:

User A converted in month 3.
User B never converted in the first year.

The naive way would be to train independent logistic models and then post-process their outputs such that the sum of probabilities is 1. However, this doesn't exploit the fact that the targets:

User A converted in month 3, user B converted in month 4.
are more similar than:
User A converted in month 3, user B converted in month 12.

Is there a modelling strategy that can help me predict when the user will convert in such a way that the sum of probabilities of converting in each month is 1? Are there alternative modelling strategies for this problem? Are there ways that exploit that targets in close months are close?


texts: Pass 2 different kinds of X training data to ML model simultaneously

texts:
I'm trying to classify if a book is fiction/nonfiction based on title and summary.

This is 2 distinct types of information - is there a way to segment title and summary before feeding it to a model, rather than concatenating the information?

For example:

Title: "such a long journey"

Summary: "it is bombay in 1971, the year india went to..."

Label: "fiction" (where fiction =1)

Current procedure:

What I've been doing until now is concatenating the information, so the above becomes,

example = "such a long journey it is bombay in 1971, the year india went to..."
label = 1

Then the usual setup, something like,

X.append(example)
y.append(label)
...
X = lemmatize(X)
...
X_train, X_test, y_train, y_test = split_data(X,y)

vectorizer = TfidfVectorizer(...)
X_train = vectorizer.fit_transform(X_train)
X_test = vectorizer.transform(X_test)

classifier.fit(X_train, y_train)
y_predict = classifier.predict(X_test)

But feeding the data concatenated feels intuitively wrong. Is there a better way to do this?

If for some reason its possible with a library other than sklearn (keras, tensorflow) I'd be also open to hearing about that.

UPDATE

Going from,

X = ['two'],['two'],['four'],['two'],['four'],['four']]
y = ['human','human','dog','human','dog','dog']

to,

X = [['two','hello'],['two','hello'],['four','bark'],['two','hi'],['four','bark'],['four','woof']]
y = ['human','human','dog','human','dog','dog']

causes errors to be thrown.

'list' object has no attribute 'lower' is X is a list, and 'numpy.ndarray' object has no attribute 'lower' if X is an array.

The error is thrown when I call,

X_train = vectorizer.fit_transform(X_train)

Is it possible to pass in a vector of features?


texts: Library to count number of objects in image?

texts: I'm reading a paper on counting cells, humans, etc. Is there an off-the-shelf library for Python/Theano or TensorFlow I could use?


texts: What does it mean to "warm-start" XGBoost?

texts:
In the project I am currently working on (predicting whether or not someone will click on some item from the mailing list that I send), each day data about users is extracted and the models are learned from scratch. Since users from yesterday don't really differ that much from users today I decided to try to somehow use the old models.

For the Deep Net and SVM models it's easy - I just make the algorithms start with the weights that were obtained last time and make them work through the new dataset using gradient descent.

What about the XGBoost though? I see that I have two options to use previous data:

Make predictions using the old model and feed them as a parameter to the new model
Use the old booster

The first one looks like it will make the new model focus too much on the old results, whereas the second one also feels a bit weird: I built 1000 trees and now I boost them with new data - so here it looks like I will end up mostly disregarding the old data and just rebuilding the forest to match the new data.

My first thought was to simply use half of trees from the old model and half from the new one. This way I would still use old knowledge but balance it out with new data. After some time I would get something like a funny pyramid-shaped forest with very few "old" trees and very many "young" ones (another way to think about it is like a power series model where data from period t-1 is given half the importance that data from period t gets). The idea looks neat, but I haven't seen anyone do that anywhere, so I remain skeptical.

Are there any options that I missed? I would be grateful for both comments on the possibilities that I outlined above and new ideas on treating this problem.


texts: Forecasting mon missing time timeseries data

texts:
I have time series data for minutes interval. But due to some noise i have to remove some rows from data. Now, I have data with some missing time stamp.
What should i do for forecasting in this case?


texts: What is the difference between a "cell" and a "layer" within neural networks?

texts:
So I understand what "layers" are.  If you have 5 layers in your model, your data basically gets transformed 5 times via 5 activation functions.  The number of "neurons" within a layer dictate how many outputs a layer creates.
So what are "cells"?  I never understood where "cells" come into play.  Are they a collection of layers?
Per Wiki: https://en.wikipedia.org/wiki/Long_short-term_memory
If the orange are layers, then I would imagine each has a bunch of neurons.  So a cell is a collection of layers and yellow stuff?  I'm having trouble understanding where this "cell" fits into an overall NN architecture.  I am used to the pictures with input layer -> hidden layer -> output layer.  So where would the "cell" occur?


texts: Should we use only one-hot-vector for LSTM input/outputs?

texts:
Should we convert our inputs to on-hot-vectors and expect one-hot-vectors as output?
I mean can we feed LSTM with a vector like x=[12, -234, 54 , 78 , 12 , 6], and have a label vector like this: y=[13, -230, 50, 80 , 9 , 7]? (And we don't use one-hot-vectors at all).
Will such network work properly? Or it's better to convert inputs/outputs to a one-hot-vector and this is essence of LSTM?
If feeding LSTM with one-hot-vector isn't a necessary rule, and we like to feed our network with such vectors in my previous question, should we again use softmax() function for out outputs? Or we can use better options for such problem(or even don't use any functions there)?
If we must(or better) to use softmax, how can we interpret it's result?
If it's better to convert our inputs/outputs to one-hot-vectors, can we use two or three hot vectors(I mean: x =[1,0,0,1,0,0] or x=[0,1,1,1,0,0])? Does this work properly or it disrupts the LSTM performance?


texts: Is there a way to automatically generate a string given an input of another string?

texts:
I am fairly new to data science in general, and I'm wondering if it is possible to use machine learning/natural language processing to generate definitions for various terms.  My team has recently been tasked with the delivery of a business glossary and using ML/NLP has been an idea that my manager wants to pursue.

I'm wondering if, for example, it is possible to write an algorithm that takes a 1-4 word string as an input and generates a 'good' definition, with good being defined by a statement of what the term is/means without being circular (e.g. the definition for 'Account' should be simply be 'Account' or 'An account').

I've been exploring various Python libraries (nltk, gensim, scikit-learn) and reading tutorials on how this could possibly be done, but as you can tell I'm a novice.  I've gone through the Natural Language Processing Fundamentals in Python course on Datacamp which is very helpful in building a basic classifier algorithm but that's all I've really been able to do.

My first idea is that I could write something which consumes various corpora.  I'm in Finance so my thoughts on the type of corpora would be the FIBO Glossary and the various business glossaries which already exist in-house.

So basically, I'm just wondering if

What I'm trying to do is even possible
If it is, if anyone here knows of a or use case where this has been implemented or even what the name of such an algorithm, function is, as that would make looking for resources easier.

Thank you for your help.


texts: The model of LSTM with more than one unit

texts:
In stacked LSTM, for example: 2 LSTM layers, LSTM_1 in order to pass the output of every time step to LSTM_2, so it needs to return hidden state value in every time step, like the architecture I drew below:

But the picture above only has one unit, so I am wondering if it has more than one unit, for example memory unit=2, what will this model look like.

or

or can someone point out the wrong part, or give a sample of visualize architecture of LSTM model with multiple units, thanks!

perhaps my description was not clear enough, so i try to give a more clear description and graph of the question:
I try to illustrate the question as the picture below, and my question is: is the architect correct when LSTM model has 2_units,2_layers.


texts: K-Means clustering for mixed numeric and categorical data implementation in C#

texts: I am a research scholar in data mining. I'm interested in C# implementation of K-Means clustering algorithm for mixed numeric and categorical data.


texts: The best practice for implementing a dynamic classification model that can have classes added to it frequently?

texts:
Let's say you are tasked with training a malware classification model, but the problem is that new malware families could be added to this model every week. for example a new malware family shows up, so you gather thousands of sample of it and you need to add this new class to your classification mode and retrain it, and this can happen every week.
So for this problem, what is the best approach to implementing it ? (In pytorch or tensorflow, doesn't matter)
Basically if you wanted to train a classification model that can have new classes added to it frequently, what is the most optimize way of implementing the model and retraining it and adding the new class to it? (The model can be a machine learning model like forest, or a deep model, this doesn't matter for me either)


texts: Is my general understanding of finding weights correct?

texts:
I started a course in Deep Learning. I'm trying to make an example in order to explain to myself how the weights are found mathematically.
If what I wrote below is nonsense I'll be glad to hear an explanation. Thanks.

So, for a given image we do WX+b. We get some vector Y and then we compare it to a desired label vector L according to . I'm assuming that we calculate D with "Cosine Similarity". For simplicity S(Y)==Y. So what we're trying to do is to calculate  so it will be one.
Let’s say we have image X of the letter “a”  and two labels (“a”, “b”). Then  . We want to calculate W and b for which we will get such vector  that when we’ll insert it into  we’ll get zero. We convert X to a vector. Since we have 2 labels and size of the X is 9, the W and b are the following: .  So, we get: . This gives us the following system of equations: . So, now we need to solve the following .

If what I wrote above is not nonsense, I don't quite understand where finding minimum is applied?


texts: how to get the classe names in an image classifer after predection?

texts:
I made an image classifier of 80 classes of handwritten numbers then I tested my model and it worked pretty fine, the only problem that I have now is the display of the correct names of these classes.
Dataset:
2 folders: [Train Folder===> 80 folders each has 110 images, Validation folder===> 80 folders each has 22 images]

Bellow the code I used for training, saving and testing my model:

from keras.preprocessing.image import ImageDataGenerator
from keras.models import Sequential
from keras.layers import Conv2D, MaxPooling2D
from keras.layers import Activation, Dropout, Flatten, Dense
from keras import backend as K

# dimensions of our images.
img_width, img_height = 251, 54
#img_width, img_height = 150, 33

train_data_dir = 'C:/Users/ADEM/Desktop/msi_youssef/PFE/test/numbers/data/train'
validation_data_dir = 'C:/Users/ADEM/Desktop/msi_youssef/PFE/test/numbers/data/valid'
nb_train_samples = 8800 #10435
nb_validation_samples = 1763 #2051
epochs = 30 #20 # how much time you want to train your model on the data
batch_size = 32 #16

if K.image_data_format() == 'channels_first':
    input_shape = (3, img_width, img_height)
else:
    input_shape = (img_width, img_height, 3)

model = Sequential()
model.add(Conv2D(32, (3, 3), input_shape=input_shape))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(32, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Conv2D(64, (3, 3)))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))

model.add(Flatten())
model.add(Dense(64))
model.add(Activation('relu'))
model.add(Dropout(0.5))
model.add(Dense(80)) #1
model.add(Activation('softmax')) #sigmoid

model.compile(loss='sparse_categorical_crossentropy',optimizer='rmsprop',metrics=['accuracy'])#categorical_crossentropy #binary_crossentropy

# this is the augmentation configuration we will use for training
train_datagen = ImageDataGenerator(
    rescale=1. / 255,
    shear_range=0.1,
    zoom_range=0.05,
    horizontal_flip=False)

# this is the augmentation configuration we will use for testing:
# only rescaling
test_datagen = ImageDataGenerator(rescale=1. / 255)

train_generator = train_datagen.flow_from_directory(
    train_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size,
    class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
    validation_data_dir,
    target_size=(img_width, img_height),
    batch_size=batch_size,
    class_mode='binary')

model.fit_generator(
    train_generator,
    steps_per_epoch=nb_train_samples // batch_size,
    epochs=epochs,
    validation_data=validation_generator,
    validation_steps=nb_validation_samples // batch_size)

model.save('testX_2.h5') #first_try

last epoche resulat

Epoch 30/30
275/275 [==============================] - 38s 137ms/step - loss: 0.9406 - acc: 0.7562 - val_loss: 0.1268 - val_acc: 0.9688

how I tested my model:

from keras.models import load_model
from keras.preprocessing import image
import matplotlib.pyplot as plt
import numpy as np
import os

#result = [10,7]

def load_image(img_path, show=False):

    img = image.load_img(img_path, target_size=(251, 54))

    img_tensor = image.img_to_array(img)
    img_tensor = np.expand_dims(img_tensor, axis=0)
    img_tensor /= 255.

    if show:
        plt.imshow(img_tensor[0])
        plt.axis('off')
        plt.show()

    return img_tensor

if __name__ == "__main__":

    # load model
    model = load_model('C:/Users/ADEM/Desktop/msi_youssef/PFE/other_shit/testX_2.h5')

    # image path
    img_path = 'C:/Users/ADEM/Desktop/msi_youssef/PFE/dataset/1.75/eeza.png'

    # load a single image
    new_image = load_image(img_path)

    # check prediction
    pred = model.predict_classes(new_image)
    print(pred)

it gives me this result instead of given the name of folder:

[7]


texts: Why my model can't recognise my own hand written digit?

texts:
Currently i am working on digit recognizer[0-9]. My model train accuracy 100% and test accuracy 90%. But when i train to feed my own written digit, it always give me wrong prediction.

I know test and train images should come from same source. But how could i feed different source data?


texts: Train classifier to detect crest of wave

texts:
In the picture they are data and target in time series. The data is padded until reach the max length. The target is marked by human. It pin down starting point and stopping point of pulse.
First idea is encode the time series to be a text like DNA sequence and then apply machine translation to it. My inspiration comes from English and German translation(aka word replacement)
Update 27 May 2021:
I found similar experiment in the kaggle
Question:
Suppose I use seq2seq neural network train on this dataset. Later on I test with same data, but shift time to right hand side 1000 points. Will it be able to detect new starting point and new stopping point?
If not
Can neural network detect the starting point and stopping point?


texts: Difference between from nltk import word_tokenize and from nltk.tokenize import word_tokenize?

texts: What is the difference between the word_tokenize, one imported directly from nltk and the other being imported from tokenize package of nltk?


texts: How is error back-propagated in a multi-layer RNN

texts:
Let's say I have a 2 layer LSTM cell, and I'm using this network to perform regression for input sequences of length 10 along the time axis.

From what I understand, when this network is 'unfolded', it will consist of 20 LSTM cells, 10 for each layer. So the 10 cells corresponding to the first layer receive the network input for t = 1 to 10, whereas the 10 cells corresponding to the second layer receive the first layer's output for t = 1 to 10. In other words, the output from the cell in layer 1 corresponding to t = 1 goes to (1) the 'next' cell in layer 1 corresponding to t = 2, and (2) the cell in layer 2 corresponding to t = 1.

So when the error is back-propagated, will there not be two derivatives coming into each cell in layer 1? If so, how is weight update performed? Is the sum or mean of both derivatives used or is there something else going on?


texts: Machine Learning algorhtm that can be trained to learn Pig-Latin

texts:
Pig-Latin, and other similair argots, were quite popular when i was young (but since i grew up in Sweden the biggest was Rövarspråket). These language games almost always consist of a given set of rules from which letters and words are translated into something else. For example in Pig-Latin the rules are the following (source):

  For words that begin with consonant sounds, all letters before the
  initial vowel are placed at the end of the word sequence. Then, "ay"
  is added, as in the following examples:[10]

  "pig" = "igpay"
  "latin" = "atinlay"

  When words begin with consonant clusters (multiple consonants that
  form one sound), the whole sound is added to the end when speaking or
  writing.[11]

  "cheers" = "eerschay"
  "shesh" = "eshshay"

  For words that begin with vowel sounds, one just adds "way" or "yay"
  to the end (or just "ay"). Examples are:

  "eat" = "eatway" or "eatay"
  "omelet" = "omeletway" or "omeletay"

  An alternative convention for words beginning with vowel sounds, one
  removes the initial vowel(s) along with the first consonant or
  consonant cluster. This usually only works for words with more than
  one syllable and offers a more unique variant of the words in keeping
  with the mysterious, unrecognizable sounds of the converted words.
  Examples are:

  "every" = "eryevay"
  "omelet" = "eletomay"

My question is if there are any Machine Learening algorithms that is suitable to be able to learn to translate a sentence to Pig-Latin by training on a large set of "english"="pig_latin"-pairs without explicity knowing anything about the rules? Or if anybody can motivate and point me to the right direction.

My goal is to be able to input an arbitrary word, which might not even be a real word, and get the "Pig-Latin"-translated word.

My initial idea was to set up a Neural Network (mainly because thats what I know best) where the input neurons should take the ASCII-code for each letter and the output should be the ASCII-code for each letter (always capital letters), i.e.

Input (for PIG):
080 073 071

Output (for IGPAY)):
073 071 080 065 089

A neural network can be trained to map for example 080 to 073, but how do i for example handle the aribtrary length of the input and output (if Neural Networks ar applicable to this problem).


texts:
This explains why we need both the sine and the cosine for each frequency: if we only used the sine (the blue wave at i   =   100), the model would not be able to distinguish positions p   =   25 and p   =   35 (marked by a cross). Géron, Aurélien. Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow (Kindle Locations 12575-12577). O'Reilly Media. Kindle Edition.

Why doesn't the model distinguish positions p = 25 and p = 35, if sine function is used only?

Isn't it possible to distinguish the positions such that the model refer to other dimension (e.g. other i, 50, 30, etc.)?


texts: How to encode a column containing both string and numbers

texts: I have a column in my dataset which contains both number and strings as the value. I want to encode the string variable so to use it for predicting. What is the best way to do this?


texts:
I joined a data science learning community in my college and we are using linux terminal commands and awk commands to practice gathering some information from big datasets stored in csv files. About 7140596 columns by 29 rows in a single file.

A sample question would be: "What was the average arrival delay (in minutes) for flights in 2005?" where we have to sum the value of the delay for every row and divide by the total number or rows.

I know that similar data manipulation can be done in Pandas in a Jupyter notebook and was wondering what are the advantages and disadvantages of each method.

Thank you!


texts: Predicting a timeseries that includes categorical and numeric data

texts:
Given the following properties in a dataset:

Type Of Work [T1,T2,T3,...Ti]
Measurment Unit [U1,U2,U3,...Ui]
Number <int>
Date <datetime>
Status [S1,S2,S3,...Si]

What can i use to predict the Status in a given Date?

Any suggestion is really appreciated.


texts:
I have few questions:

Is there a website to upload huge research dataset (over 100GB) for free?
Which type of compression (rar, zip ... etc) is good for jpeg images?
In case of dataset of 120GB. what is the best split for this big files (eg: 20 GB each)?