texts: Need equations for some of weight initializers in tensorflow?

texts:
For my CNN model I tried some of the weight initializers such as "truncated normal initializer","random normal initializer","glorot normal initializer","glorot uniform initializer", my questions are:

1- I need the equations for each of them, for the two Xavier initializers I found more than one type of equation for that I confused which one is true but for other two I didn't found any equation?

2- Is the range of values are between 0 and 1 for all of them? are they considered a gaussian?


texts:
I know that is better avoid loop in Keras custom loss function, but I think I have to do it.
The problem is the following: I'm trying to implement a loss function that compute a loss value for multiple bunches of data and then aggregate this values in an unique value.

For example I have 6 data entry, so in my Keras loss I'll have 6 y_true and 6 y_pred.
I want to compute 2 loss value: one for the first 3 elements and one for the last 3 elements.

Example of hypothetical code:

def custom_loss(y_true, y_pred):
    start_range = 0
    losses = []
    for index in range(0,2):
        end_range = start_range + 3
        y_true_bunch = y_true[start_range:end_range]
        y_pred_bunch = y_pred[start_range:end_range]
        loss_value = ...some processing on bunches...
        losses.append(loss_value)
        start_range = end_range
    final_loss = ...aggregate loss_value...
    return final_loss

Is it possible to achieve something like this? I need to process the whole dataset and calculate loss for multiple bunches and then aggregate all bunches value in a single value.


texts: statsmodels cointegration test order of variables

texts:
I am testing two time series for cointegration as follows:
from statsmodels.tsa.stattools import adfuller, coint

coint(x, y, maxlag=1, trend='c')
>>> (-4.164121561476605,
>>> 0.00413161869902322,
>>> array([-3.92382237, -3.35133355, -3.05499082]))

coint(y, x, maxlag=1, trend='c')
>>> (-3.310399938372457,
>>> 0.053322513769096906,
>>> array([-3.92382237, -3.35133355, -3.05499082]))

I did not expect that reversing the order of the two 1d series would change the result of the test for cointegration. In my case, x and y are close prices for gold ETF GLD and miners GDX respectively, so the order of variables seems arbitrary. What is the proper way to interpret the differing results?


texts: How can one determine that Word2Vec (CBOW method) embeddings are related to each other?

texts:
I read some fascinating stuff about the potential for using the Word2Vec algorithm to speed up the pace of scientific discovery here https://www.researchgate.net/publication/334209824_Unsupervised_word_embeddings_capture_latent_knowledge_from_materials_science_literature. In the paper, the researchers trained the algorithm on millions of materials science abstracts to identify potential associations that human researchers had missed.

My question concerns using the Word2Vec in contexts that are more disparate than a single scientific field. The CBOW method takes in an input of words and returns a predicted word that would most likely be associated with it. So if I had trained the algorithm on economics papers, an input of externalities, public good, and tragedy of the commons might return a prediction of carbon pollution. However, had I trained on ecology papers and used the same input, I would not get the same output, and the input might well be meaningless due to the lack of those words in the training set. The question then becomes, how can one tell if the embedding of a word in one field, is similar to that in another field? The application I am imagining for this is the potential to solve problems in one field using thought processes and methodologies already developed in another field. Would words with similar vectors in different field automatically indicate a connection between them, at least in the structure of the words around them?


texts:
Objective

I want to visualize a typical customer's journey using Python or R. As usual, customers buy different products at different points of time.

I checked every customers journey and chose product A as a starting point. 10,000 customers bought product A as their first product. Now I want to show the "flow" of their  journey. After 10 minutes, 2,500 of these 10,000 customers,  bought product B as their second product. Product C was bought by 7,500 people after 25 minutes.

I am looking for a more complex sankey chart. You start of with a bar chart with the height of 10,000 on the left. There are two bar charts on the right and you see a connection between the left and right charts. The longer it took the customers to buy the next product, the further away are the bars on the right from the left one. All the bars have titles with their product names. Quick infos on the charts would also be nice.

The final chart would visualize a longer customer journey (more than two products).

What I have tried

I used the networkD3 R package to visualize a sankeyNetwork, however I could not place the bar charts according to my desires.

Do you have any recommendations for Python or R packages, or is there a way to draw it by myself (in a reasonable amount of time)?


texts: Quantify the amount of differences of one data set to another

texts:
Basically I'm seeking the best method for my case scenario here. For example with the image shown below, the left one is poincare maps dataset #1, and the right is dataset #2.

What's the best method of quantifying the differences of the right one to the left? Standard deviation won't be of much help here, since these are two separate data sets. Appreciate any inputs, thanks in advance!


texts: How to approach different image resolutions in deep learning for regression problem?

texts:
I have an image dataset of various resolutions and using regression DNN model with fixed n*n input resolution. As model learns certain positions in the image, I've been using zero padding to fit images resolutions to maintain 1:1 aspect ratio.
Is there a better way to preprocess images?
Without zero padding, I get worse results, and I guess maintaining aspect ratio is necessary to avoid objects' shape distortions in DNN input.


texts: Unable to understand Kernel Ridge regression

texts:
I am trying to read kernel ridge regression from this link

But , I am unable to get the intution behind the derivation.

Can anyone please help me ?


texts:
I have to find make a classifier for price prediction of a item. The question I have is which columns I should choose for price prediction.

Also which machine learning classifier would be good to perform this, at present I choose random forest.

Do I need to use time series concept in here?, I think No


texts:
I have 4 standard normal features on which I perform PCA. I then take the first principal component (with all of the components). Is it possible to a priori say what is the max and the min value that the transformed series will have?

I guess, if we assume that the original standard normal features never exceed +/- 5, then the max of the final transformed series would be the PCA coefficients product summed by 5? But is it possible that the PCA coefficients are [1, 1, 1, 1] in this case? Or is there some sort of a bound on what the PCA coefficients can be?


texts:
I have a problem where I want an LSTM to predict the resistance of a body. This value can also be calculated if we know the drag coefficient and the speed of that body. In my case, at inference time, the speed is known, meaning that I can do the following:

predict the drag coefficient, and then calculate the resistance accordingly
predict the resistance directly

Which one should I use as my learning target?


texts: Is adding the embedded words of a sentence to represent the sentence a good approach?

texts:
I have a dataset of sentences in a non english language like :

word1 word2 word3 word62

word5 word1 word2

Now i want to turn each variable length sentence to a fixed size vector to give it to my model, and i want all the words in the sentences to have effect on the output
I thought maybe i can use an algorithm like word2vec and turn each word into a fixed size vector, and add all of them to represent the sentence, is this a meaningful approach? is this better than adding the hot one vectors of the words to represent the sentence?  is there a better approach than these two?
EDIT1: basically i have a dataset of random variable length sentences and i want to embed them the best way possible, meaning keeping as much  information as possible in the resulting embedded vectors (which all have the same size)


texts:
To data scientists who use python for development, how do you demo your work?

I used to demo my work in Shiny as my developments was mainly in R. Now I develop 99% in Python but not sure of the best way to demo work (predictive modelling, data analytics side, pattern detection,..etc)?

Currently I use Dash, but the amount if time I spend on html and css makes it less appealing.


texts:
I have several different scores that I'd like to combine them into a single score that represents all parameters. Score components are:

Nutrition Score (1-10): 1 for the worst quality and 10 for the best quality
Processing quality (1-4): 1 for the least processed and 4 for the most processed food
Some other features with different scaling

Overall score = w1 * Nutrition_Score + w2 * Processing_Quality + w3 * parameter_3 + ...

Considering that I don't have a mean to measure and distinguish the importance of each of these parameters compared to the rest of them, is there any way to assign weights to each of them except the heuristic way of doing it?


texts: Role and need of Mathematics in Data Science

texts: I am Bachelors of Engineering in Software Engineering field and having 3 years professional work experience in Oracle ERP/Database related technologies. Now I'm planning to pursue MSc in Data Science. What I need to know that what sort of Mathematics skills should a Data Scientist have? As we had learned numerous subjects like Numerical Analysis, Statistics, Linear Algebra etc.


texts: What is the closest interpolation method to INTER_AREA in KERAS for shrinking images

texts:
What is the closest interpolation method to INTER_AREA in Keras from the following:
nearest, bilinear, bicubic, hamming, box, lanczos.
I always used INTER_AREA for shrinking images but for some reason, it is not supported in:
https://keras.io/api/preprocessing/image/#flowfromdirectory-method
in the "flow_from_directory" method.


texts: Why do seaborn.dist and pyplot.hist generate two different looking histograms on the same data?

texts:
I'm looking at telecom customers data. Two of the variables I'm looking at currently are:

Monthly Charges - The total amount charged to the customer monthly.
Is Senior Citizen - Whether the customer is a senior citizen.

I'm trying to plot two histograms to see if the distributions for non-senior and senior citizens is different.

If I use seaborn's distplot then I get the following result

And if I use pyplot hist then I get the following result

In the first plot the blue one towers above the orange ones in the range ~70-120 whereas in the second image the blue one always stays below the orange one.

What is the difference between these two?


texts: Convolutional Neural Networks - When do I adjust parts of it, and how?

texts:
I'm still training a Convolutional Neural Network model in Tensorflow to recognize age groups from facial images. I'm using the Adience 3D (frontalized with a 3D model - it might not be good data) dataset, with four folds for training, and one fold for testing.

I'm currently using three Conv-Conv-Pool layer sets, the first one's Conv layer using 32 filters with 5x5 size, the second and third using 64 filters with 3x3 size, and to top it all off, a Dense layer with flattened input, then a Dropout layer, dropping out the outputs of the dense layer by 20%, then on to the logits layer.

I normalized my cropped color images with the per-channel normalization - the (A - np.mean(A)) / np.std(A) method for each 64x64 array in the 64x64x3 input.

Currently I'm training it with Batch Size 64 and 100 steps so as to evaluate the weights every 100 steps, and while I don't see much improvement for now, I could see it increasing from steps 100 to 500 (0.27 to 0.35), before dropping on stage 600 then going up again up to step 800 (0.30 - 0.34).

Is this a sign I have to change my layer structure?

On that note, when should I change up my layer structure/hyperparameters like training steps, batch size, filter size, etc., and how?


texts: Chossing between gradient boosting algorithms

texts:
I just stepped in machine learning competitions and it looks like most of the mid-sized dataset competitions are won by Gradient boosting based models. However I came accross case where LightGBM,Catboost or Adaboost had very different scores.
Is there a method to choose between those algorithms?


texts: Is a new datapoint similar to my reference set?

texts:
I have a large reference data set and everyday I get new data points.
I would like to know how can I test if those new data points are similar to the reference set?

Thanks in advance.

Maria


texts: How often do we see normally distributed data

texts:
I am having difficulty in exactly understanding several statistical tests, such as the t-test and ANOVA test. These tests require that the data we use be normally distributed.

However, whilst sharing my experience in analyzing data a bit, I have analyzed several data sets from numerous sources online (web scraping, open-accessed data sources online, etc.), with the considerably high number of samples (hundreds, thousands). An example of the data in question is the amount of donation given to certain campaigns in a fixed periods of time (day 1 at 1pm, day 2 at 1pm, etc.).

And when I tested whether the distribution of the data was normal, using visual aids (histograms, Q-Q plots) and Shapiro-Wilks test, they all showed me that the data is not normal. For example, Shapiro-Wilk test gave a p-value of so small (less than 0.00000000000000022), of course, the null hypothesis has to be rejected, i.e. the data is not normally distributed.

Because I read in articles like in this link, it says

  However, even if the distribution of the individual observations is not normal, the distribution of the sample means will be normally distributed if your sample size is about 30 or larger

So naturally, I am confused, is my data normally distributed or not? How often do you encounter normal and not-normal distribution, in real-life data?

EDIT
Following the response by @hssay in his answer and comments, my main objective here is that I want to do ANOVA test to determine the relationship between my numerical and categorical data. But ANOVA needs the data to be normally distributed. So now I am confused as to how to conduct it since I have a "sample" consisting of thousands of rows of data that I took only once.


texts: How to integrate nvidia gpu with jupyter notebook for scikit-learn

texts: I know there are answers saying that tells how to integrate/use tensorflow or for deep learning libraries. I want to use it for classification or regression scikit-learn libraries. Is it possible. If yes then how ?


texts:
I’m reading the paper Spectral Networks and Deep Locally Connected Networks on Graphs and I’m having a hard time understanding the notation shown in the picture below (the scribbles are mine):

Specifically, I don’t understand the notation for the matrix F. Why does it include an i and a j?


texts: How to calculate cov(e|X) in Generalised linear regression?

texts: This image taken from wiki https://en.wikipedia.org/wiki/Generalized_least_squares. I am not able to calculate Cov(e|X)? Can someone tell me that.And also give me dimesion of cov(E|X)= sigma^2 * rho


texts: What Predictions can be done for Restroom/Washroom Data?

texts:
My company has installed sensors, which actually monitors the Restrooms/Washrooms in the Building. Currently the sensors collect data such as Ammonia, Nitrous and Visitors Count;

All the data collected are time-series data and I have a huge quantity of those. My question is what type of predictions can be done that can add value to this business?

Can I use the power of Data Science to avail something which can make my restroom more smarter?


texts: Z values get rescaled while plotting in 3d

texts:
x_3d = 5.171875  , 5.3125    , 5.171875 ...  6

  y_3d  = 0,   1,   2,   3,   4,   5,  .... 1500 (1500 points)

  z_3d = -10.493983  ,  22.128998  , -14.582531  ,  34.687847  ,
          11.390786  ,   9.852596  ,  -4.8797636 , -18.590075  ,
          10.95118   , -19.660675  ....(1500 points)

Here

x_3d lies between 5 & 6 (1500 points)

y_3d lies between 0 & 1500(1500 points)

z_3d lies between -100 & 100(1500 points)

But somehow on plotting the 3d scatterplot z values get diminished around 0

I am using the code

plt.figure(figsize=(16,10))
ax = plt.axes(projection='3d')
ax.view_init(0,90)
plt.scatter(x_3d,y_3d,z_3d,c=z_3d,cmap='viridis')
ax.set_zlim(-100, 100)

In the plot somehow z range is changing and settling around 0
I am very confused please help

Thank you in advance


texts: Can Google really bring back billions of results in a blink of an eye (almost)

texts:
I was always fascinated by Google's search ability, a great achievement by Google and other search engine providers also, but more so a collective human talent and ability that makes me appreciate our amazing mind and our potential to innovate.

I use Google search daily and I am sometimes disappointed with some very few words that would give no results which I accept to an extent. One of these instances led me to do a further "mini" investigation/test on Google's search ability. For example I tried searching for the word "packit4me" (www.packit4me.com) that I was trying to make use of as an api in as system I was developing. When I searched for "packit4me" no results came back at all even though the site was derived through Google search initially. OK fair enough it isn't all that perfect especially with a word like that. I then decided to test how far I could push its limits. My next test led me to question "did Google manage to do what it said it just did in a flash of a second". I always believed that it is either bringing back pointless results after a few pages or it is just amazingly sophisticated with the amount of data it is filtering.

I'm the sort of person that sometimes likes to put applications to test even if it means trying something that sounds pointless.

So to put Google to the test I searched for "liverpool vs real madrid" it said:

  "About 281,000,000 results (0.53 seconds) "

I then decided to go up to as many pages as possible to see how the accuracy in search results changes (not that I am all out doubting it). I wanted to approach it with a test in strange way that even I had never tried before.

So believe it or not I got to the last link in the Google search list which was:

  "repeat the search with the omitted results included."

You would think I sound like I will be going for hours but that was only at about page 14.

So I clicked the link and carried on...

I carried on until I got to the very last result which was on results page 48 and which was no were near where I thought would have given up and stopped at. It was however no were near the initial message claiming some 280m results. Unfortunately because I wanted to get to a really good distance to test the accuracy (and because it got a bit boring) I was skipping in a speedy manner so I didn't pay much attention to how accurate it was with the page results as I paged through.

Anyway I scrapped the first test and was more surprised by the fact that after a few page skipping I got to the very last result of what was initially meant to be some 280m results. The actual total number of results that Google only managed to bring back was 471 results.

  "Page 48 of about 471 results (0.97 seconds) "

Basically my question is why doesn't the results message reflect the actually results that can or has actually been displayed to me? By far it is just somewhat way way too off.

I suppose the pressing question to me is that with the advancement of the technology we have these days is it really possible to bring back some billions and billions of results from the vast network of the web. First of all I think it is pointless to show this amount of data and the same applies to claiming it is there on Google's part.

To me it's either a bug or miss information in the example of Google, or I suppose I can maybe raise the question can this sort of task be carried out in such a vast network in a split of second or are we still a long way off the technology and power machines required to bring back this volume of results at such speed and accuracy?


texts: There are so many CNNs out there and I am trying to do a comparison between some of them in my paper. Which networks should I include? Resnet, VGG, and Inception are obvious, but I would like three or four others. Which ones should I choose? There are so many variations out there like Xception, resnext, and ... that I am confused. Which of these networks are most used in the academic papers?


texts: Frequency bin of the positive and negative frequency

texts:
I am using this MATLAB tutorial for Frequency-Domain Linear Regression. There is one part of code they provide, where it is necessary to determine the "frequency bin of the positive and negative frequency", for fft and ifft:

freqbin = 72/12;
freqbins = [freqbin 72-freqbin]+1;
tsfit = zeros(72,1);
tsfit(freqbins) = tsdft(freqbins);
tsfit = ifft(tsfit);
mu = mean(ts);
tsfit = mu+tsfit;

Length of time series is 72, and 12 months is considered as one cycle. How there can be only one frequency bin for positive frequency? How do we know that it is exactly 72/12 + 1 (+1 is because first bin is for zero frequency)? Is this some formula or what?

I don't understand what is exacty frequency bin, and why there is only one bin for positive frequency? I thought that we have many bins in time series.


texts:
In Stata, I can perform a conditional replace using the following code:

replace target_var = new_value if condition_var1 == x & condition_var2 == y

What's the most pythonic way to reproduce the above on a pandas dataframe? Bonus points if I can throw the new values, and conditions into a dictionary to loop over.

To add a bit more context, I'm trying to clean some geographic data, so I'll have a lot of lines like

replace county_name = new_name_1 if district == X_1 and city == Y_1
....
replace county_name = new_name_N if district == X_N and city == Y_N

What I've found so far:

pd.replace  which lets me do stuff like the following, but doesn't seem to accept logical conditions:

`

replacements = {   1: 'Male',   2: 'Female',   0: 'Not Recorded' }

df['sex'].replace(replacements, inplace=True)

`


texts:
I'm in the progress to learn, and understand different neural networks. I pretty much understand now feed-forward neural networks, and back-propagation of them, and now learning convolutional neural networks. I understand the forward-propagation of them, but having issues understanding their back-propagation. There is a very good resource explaining the convolutional layer, however, can't understand the back-propagation.
In my understanding, according the back-propagation algorithm of feed-forward neural networks/multi-layer perception, if I have the following input (its items as $i$), and filter (its items as $w$), giving the output (its items as $o$).
$$\begin{pmatrix}i_{1}^1 & i_{2}^1 &  i_{3}^1\\\ i_{4}^1 & i_{5}^1 &  i_{6}^1\\\ i_{7}^1 & i_{8}^1 &  i_{9}^1\end{pmatrix} * \begin{pmatrix}w_1^1 & w_2^1\\\ w_3^1 & w_4^1\end{pmatrix} = \begin{pmatrix}o_1^1 & o_2^1\\\ o_3^1 & o_4^1\end{pmatrix}$$
So if we want to calculate for example how much $w_1^1$ affected the cost $C$, we need to know how much $w_1^1$ affected its corresponding output item $o_1^1$, and how much $o_1^1$ affected the cost $C$ which gives the following equation:
$$\frac{\partial C}{\partial w_1^1} = \frac{\partial o^1}{\partial w_1^1}\frac{\partial C}{\partial o^1}$$
Where in my thinking we have to think back how we get the output regarding to $w_1^1$ to calculate $\frac{\partial o^1}{\partial w_1^1}$.
To get $o_1^1$, we multiplied $w_1^1$ with $i_1^1$, to get $o_2^1$, multiplied $w_1^1$ with $i_2^1$, to get $o_3^1$, multiplied $w_1^1$ with $i_4^1$, to get $o_4^1$, multiplied $w_1^1$ with $i_5^1$.
To calculate $\frac{\partial C}{\partial o^1}$, it depends on how the output is connected with the next layer. If it is an another convolutional layer, then we have to calculate how each output item is connected to the next layers outputs, which will be their connecting weights.
So if we see an example, where we put a 2x2 filter on $o^1$, to get the final output $o^2$ (which will give a single output with 1x1 size):
$$\begin{pmatrix}o_1^1 & o_2^1\\\ o_3^1 & o_4^1\end{pmatrix} * \begin{pmatrix}w_1^2 & w_2^2\\\ w_3^2 & w_4^2\end{pmatrix} = \begin{pmatrix}o_1^2\end{pmatrix}$$
Where in my thinking the back-propagation for $w_1^2$ is:
$$\frac{\partial C}{\partial w_1^2} = \frac{\partial o^2}{\partial w_1^2}\frac{\partial C}{\partial o^2} = o_1^1 * 2(o^2_1 - y_1)$$,
and the back-propagation for $w_1^1$ is:
$$\frac{\partial C}{\partial w_1^1} = \frac{\partial o^1}{\partial w_1^1}\frac{\partial C}{\partial o^1}$$
Where: $$\frac{\partial o^1}{\partial w_1^1} = (i_1^1 + i_2^1 + i_4^1 + i_5^1)$$
And: $$\frac{\partial C}{\partial o^1} = \frac{\partial o_1^2}{\partial o_1^1}\frac{\partial C}{\partial o_1^2} + \frac{\partial o_1^2}{\partial o_2^1}\frac{\partial C}{\partial o_1^2} +\frac{\partial o_1^2}{\partial o_3^1}\frac{\partial C}{\partial o_1^2} +\frac{\partial o_1^2}{\partial o_4^1}\frac{\partial C}{\partial o_1^2}$$
So: $$\frac{\partial C}{\partial o^1} = w_1^2 * 2(o_1^2 - y_1) + w_2^2 * 2(o_1^2 - y_1) + w_3^2 * 2(o_1^2 - y_1) + w_4^2 * 2(o_1^2 - y_1)$$
Am I right? Because as I'm reading through the article above, it seems completely different.


texts: word2vec -storing a word and its vector yet be able to efficiently run k-nearest neighbour

texts: After training a model using word2vec I'd now like to store the trained model with the word serving as a key and the vector as its value. However I'm not sure how I'll be able to implement in this way a k-nearest neighbour search. What would be the correct way to get around this?


texts: Can I get some advices on inferencing people from upwards using Yolov5?

texts:
I'm trying to inference people from upwards and count them using Yolov5.
I know the controversy between yolov5 and yolov4, but for me, Yolov5 is more easier and reliable to use, also the setup.
I have tried SORT, Deep-SORT to track count people passing the gate, but it gets lost when the camera directly films from the upper side, so the tracking gets lost and maps with new id.
When I remove the tracker I can see that the inferencing is not happening.
I'm not sure I can solve this by touching the script, or the only way is to
train the model. Currently I have been using pretrained model, which I believe is been trained by COCO dataset. I have been using the people class only from the model.
If I have to train the model, can someone give me guidance about how to do that?
I am wondering how to train, as the pretrained model is containing 80 classes, is it better to wipe out everything and train it from scratch, or to use Transfer-learning tool. And if I decide to use Tlt, should I keep the 80 classes, or only left people class? I'm using the people class only, but wiping out the other classes would affect the whole network, from my understanding.
Thanks in advance!


texts: Does the output of the Sequence-to-Sequence encoder model exist in the same semantic space as the inputs (Word2vec)?

texts:
Does the output generated from the LSTM encoder module exist in the same semantic space as the original word vectors?
If so, say for example we have a sentence and we pass it through the encoder to get an encoded output and then we also calculate the average of word vectors for the same sentence separately, will the two new vectors (encoded and average) be comparable? Will their euclidean distance be relatively small?


texts: Image Embeddings Layer with Pre-Trained Model

texts:
I have some basic experience with embedding layers in NLP. However I would like to explore how can I use embedding layers to compare similarities between images. There are a few questions I am stuck with.

Where should I place my embedding layer? My guess is that it should be after the pre-trained model followed by a GlobalAveragePooling layer and then a output layer (See below). I tested this but my accuracy and loss seem to be stuck at very poor values across 200 epochs.

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
efficientnetb7 (Functional)  (None, 2560)              64097687
_________________________________________________________________
embeddings (Embedding)       (None, 2560, 128)         12800000
_________________________________________________________________
global_average_pooling1d (Gl (None, 128)               0
_________________________________________________________________
dense (Dense)                (None, 10)                1290
=================================================================
Total params: 76,898,977
Trainable params: 76,588,250
Non-trainable params: 310,727
_________________________________________________________________

Since I am using RGB images, does that mean that I should use GlobalAveragePooling3D instead of GlobalAveragePooling1D?

I have looked up at rom1504's example but couldn't seem to understand how he did his embedding layer (He seemed to have just used the pretrained model without adding an embedding layer). Some people here have also suggested using Autoencoders but I don't think that is suitable for my application. SAS Viya has a wonderful guide on image embedding using the "sasdlpy" library. However, I am wondering if this can be done with just Keras. Also his usage of triplet models might be too advanced for me.


texts: Neural network regression is not dynamic enough to predict target range?

texts:
I'm working with a CNN on a regression task, using MSE as the loss function. Here's a plot of predictions vs targets for the training set.
Note: Legend is wrong. Blue = prediction vs target | Red = fit | Green = ground truth (x=x)

I know that the approximate mean for the target is 0.8 so I initialise the bias of the last fully connected layer to this. (In my case it's reasonable to assume this for unseen samples as well). That's why the predicted fit crosses the ground truth at (0.8, 0.8).
What's interesting is that the R^2 is pretty good, but the slope is off. If I wanted to calibrate my model without further training I could easily apply a linear transformation to change the slope (note that the validation data doesn't look too dissimilar).
I feel like the model should have learned this extra factor. What could cause this lack of "sensitivity" or "dynamism"?
For the signal processing people here, this reminds me a lot of when a PID regulation loop doesn't have a strong enough actuator to keep up with fast transients. Although it's a completely different concept, I feel like somehow the weights aren't large enough to produce the necessary gradient???


texts:
I am trying to better understand the formula model for two sample t-tests in R. When I calculate the test in the formula model I get a wrong result.
set.seed(41)
df = data.frame(x1=c(rep(1, 10), rep(0, 10))+ rnorm(20, mean = 0, sd = 0.1),
               x2=c(rep(0, 10), rep(1, 10)))

t.test(x1 ~ x2, data=df)

Output
    Welch Two Sample t-test

data:  x1 by x2
t = 22.365, df = 17.85, p-value = 1.668e-14
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 0.9247087 1.1165780
sample estimates:
mean in group 0 mean in group 1
      1.0530115       0.0323681

If I use the variable model, I get the expected result.
t.test(x = df$x1, y = df$x2)

Output
    Welch Two Sample t-test

data:  df$x1 and df$x2
t = 0.2581, df = 37.945, p-value = 0.7977
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -0.2921655  0.3775450
sample estimates:
mean of x mean of y
0.5426898 0.5000000
```


texts:
In simple terms, what are the assumptions of Linear Regression?

I just want to know that when I can apply a linear regression model to our dataset.


texts: Extract sentences from beginning of news in single document summarization

texts:
I am working on Single Document Summarization task on News datasets. I do some experiments in this task. A simple experiment that I make and has a good result is extracting sentences just from beginning of news. Now I want to find any paper or research result about this type of sentence selection.

Is there any research to show how good is to choose sentences just from beginning of text without any reordering?


texts:
I have big table in dataframe (600k rows) which has y column (the variable I want to predict) and other 4 other columns that are the X.
I have run RF regressor  and  I got score of 0.87 when I run it on the train and test.
However, when I tried to predict another set of data (which is very similar, with 1M rows) I got score of 0.65.
So I assumed that is overfitting.
when I tried to understand why it hapenns, I went back to the distribution of the y column, which looks like this:

my question is, can it be that because that my data does not have normal distribution (or very skewd...) my model preformance is bad? Do all variables need to have normal distribution? how does the score of the random forest regrssion is calculated? id value is 0.25 and  predict is 0.26 does it count as correct prediction?


texts: How can I find if it is an overfitting problem?

texts:
I am new in Machine learning, and I want to detect emotions from the face.
Preprocessing: I used equalizeHist to equalizes the histogram of grayscale images (JAFFE database with 213 images), in the goal to normalizes the brightness and increases the contrast of the image.
Feature extraction: I extract features with Gabors filters from images, and I get a matrix of 213x120. I split data: 60% train data and 40% test data, and I normalize it.
Train and test: for training the model, I use SVM classifier with an RBF kernel. With grid-search, I select the best couple C, gamma (using 10-fold cross-validation). Then, I test the performance of the model on the test data (unseen data), and I get 89% accuracy.
The problem is when I want to predict emotion from a new input face, I get a false result.. Is it an overfitting problem?
UPDATE 1: feature extraction
gf = GaborFilter(ksize=(11, 11), freq_nbr=5, or_nbr=8, lambd=4, sigma=8, gamma=5, psi=5*np.pi/6) #Gabor filter with *cv2.getGaborKernel*

data = []
for j in range(len(roi1_images)): # len(roi1_images) = 213
  vect1 = feature_extraction_roi_avr(roi1_images[j], gf.kernels) # feature vect from left eye
  vect2 = feature_extraction_roi_avr(roi2_images[j], gf.kernels) # feature vect from right eye (40 features)
  vect3 = feature_extraction_roi_avr(roi3_images[j], gf.kernels) # feature vect from mouth (40 features)
  vect = np.concatenate((vect1, vect2, vect3), axis=None) # feature vect of one face (40 features)
  data.append(vect)
data = np.array(data) # Data matrix (213, 120)

UPDATE 2: Learn and test model

clf = GridSearchCV(estimator=SVC(kernel='rbf'), param_grid=svm_parameters, cv=10, n_jobs=-1) # grid search with 10fold cross validation
scaler = StandardScaler()

# Split data
X_train_, X_test_, y_train, y_test = train_test_split(data, data_labels, random_state=0, test_size=0.4, stratify=data_labels)

X_train = scaler.fit_transform(X_train) # Normalize train data
clf.fit(X_train, y_train) # fit SVM model
X_test = scaler.transform(X_test) # Normalize test data
score = clf.score(X_test, y_test) # calculate mean accuracy
print(score) # score accuracy = 0.8953488372093024

UPDATE 3: predict new input
    landmarks, gs_image = detect_landmarks(path_image)
    roi1_image = extract_roi(gs_image, landmarks, 1) #extract region 1 (eye 1)
    roi2_image = extract_roi(gs_image, landmarks, 2) #extract region 2 (eye 2)
    roi3_image = extract_roi(gs_image, landmarks, 3) #extract region 3 (eye 3)
    vect1 = feature_extraction_roi_avr(roi1_image, gf.kernels)
    vect2 = feature_extraction_roi_avr(roi2_image, gf.kernels)
    vect3 = feature_extraction_roi_avr(roi3_image, gf.kernels)
    vect = np.concatenate((vect1, vect2, vect3), axis=0)
    vect = np.reshape(vect, (1,-1))
    vect = scaler.transform(vect)
    class_ = clf.predict(vect)[0]
    print(class_,end=" ")
```


texts:
I am a bit perplex from the transformer-XL architecture that is claimed to solve the issue of context fragmantation. I probably understood it wrong but it looks like all the transformer-XL is doing, is being 2 times larger than the normal one, and that you translate at each iterations. Plus a fancy new way of doing positional encoding.

Am I interpreting it wrong ?


texts: Punctuation removal effect on n gram detection for keyword generation

texts:
I'm building ngrams after removing punctuation and lemmatization. The algorithm is to detect keywords in large bodies of text.

I have concerns that 2 documents

  The child played with the red ball.

And

  The sign was red. Balls are the toys of children.

would both contain "red ball". Are there best practices here? Ideally I would not want the second document to have the same value of "red ball" as the second.


texts: How to validate recommender model in healthcare?

texts:
In order to validate a recommender model, a usual approach is create a hold-out set that will provide random suggestions (similar to an A/B testing setup).
However, in healthcare applications, this cannot be possible as a random suggestion can put at risk a patient's life.
Hence, what is a reasonable approach to validate the model?


texts: Better code completion in python notebooks

texts: How come python code suggestion is awful in notebooks and spyder? Smetimes it shows classes methods and variables, sometimes not


texts: How to handle data with dependency on two different dates

texts:
I am currently dealing with a dataset that contains multiple date-time fields:
"buy-date" and "receive-date" which both have an effect on the prices and amount of offers made.
One example could be:

+------------+--------------+-------+--------+--------+
|  Buy-Date  | Receive-Date | price | volume | orders |
+------------+--------------+-------+--------+--------+
| 2016-07-13 | 2016-09-01   |   123 |     10 |    100 |
| 2016-07-13 | 2016-10-01   |   444 |      8 |     30 |
| 2016-07-13 | 2016-11-01   |   234 |      0 |      0 |
| 2016-07-14 | 2016-09-01   |   789 |      5 |     12 |
| 2016-07-14 | 2016-10-01   |   321 |      6 |     54 |
| 2016-07-14 | 2016-11-01   |   543 |     22 |     50 |
| 2016-07-14 | 2016-12-01   |   654 |     12 |      4 |
+------------+--------------+-------+--------+--------+

I wanted to use this data either in conjunction with an autoencoder or lstm or cnn to search for errors in the dataset.
But I don't know how to handle a dataset with two dates that influence the values.

I thought of just indexing everything from 0 to N and handle this as a normal sequence instead of a time-series. I might just use Buy-Date and Receive-Date simply as features for network instead of as indexes. E.g. convert 2016-07-13 to 3 Fields with "buy-year", "buy-month", "buy-day".

But how should I than go over the data? Can I still use a sliding Window? Should I just go through everything as if it were another kind of sequence without a temporal aspect?

Could that work or is there something I don't see? Is there a name for this type data-set? It doesn't seem to be a normal time series but it also isn't a panel.


texts: Where does BERT fit in the Machine Learning Hierarchy?

texts:
I am a newbie in the machine learning world and I need guidance from the professionals. I am trying to make a hierarchy starting from machine learning, then to deep learning and to BERT. I have read several articles but I'd get lost in the middle, I don't understand where BERT would fit in the machine learning Hierarchy.
Apologies if I my question is vague, I am more than happy to explain the part you don't understand.
Thank you.


texts: Develop classification model, given the k nearest neighbors

texts:
I have a data set listing attributes of clients - a combination of ordinal, categorical and interval data. I am also given the 5 nearest neighbors within the data set for and have to essentially reverse engineer an algorithm for the nearest neighbors.

Given the complexity of the data I'm quite lost as to how I can work out and then obtain parameters of a distance function. I apologize in advance for the vagueness of the question but I'm not even sure where to begin.


texts: Machine Learning in Paper making machines

texts:
The company I work for produces packaging paper on a mass scale; my task is to see if machine learning would be able to help identify paper breaks on the machines before they actually happen.  We have sensors and high speed camera systems on the machine for real-time analysis.

I do not know which machine learning algorithms would be applicable for this problem?

In the paper making process, a continuous sheet of paper is formed and fed through a series of machines.  If the paper breaks, the process must be restarted.  We have machines for detecting a paper break after one has already occurred, but my goal is to use machine learning to detect a break before it happens.

Could an algorithm be created that could constantly analyze many different points on the machine and based on that data be able to predict a sheet break or not?


texts: Return the most relevant columns for a given keyword

texts: Suppose my database has column name and description for each column of each table. Need to design an interface where a user can enter a keyword and the interface will return the most relevance columns. For example, enter "payment" and the interface can just find and return a list of columns containing "payment" in column name and description. But what if the user enters a keyword doesn't exist in column name and description, then how to suggest the most relevant columns for this keyword? Any suggestions?


texts: Distance between two time-dependent distribution

texts:
In my research, I want to use a meaningful and computationally tractable distance between two time-dependent probability distributions. For stationary distributions, several distance measures are used, such as KL divergence, JS divergence, Wasserstein distance, and so on.
Is anyone aware of a similar thing for two time-dependent distributions p1(x,t) and p2(x,t)?
In my problem, both distributions are the solutions of non-stationary Fokker-Planck equation,
dp1/dt = (-d/dx F1(x) + d2/dx2 D1(x) ) p1(x,t),  p1(0,t)=p10(x)
dp1/dt = (-d/dx F2(x) + d2/dx2 D2(x) ) p2(x,t),  p2(0,t)=p20(x)

Where the drift forces F1(x), F2(x), diffusion coefficients D1 and D2, and initial probability distributions p10(x) and p20(x) are computed from data. So, ideally, I want some kind of norm that can be expressed through the parameters of the Langevin equation.


texts: Is it possible to train a model on a more complicated data set and then retrain on a simpler dataset that has continuous frames for LSTM model?

texts:
I'm attempting to do lane detection using CNN-LSTM architectures on the TuSimple dataset. However the TuSimple data set isn't very difficult and the results are sometimes poor (visually). I want to train the model on a more difficult dataset such as CuLane or BDD initially but then use TuSimple for the LSTM section as it is the only dataset that has all of its data in continuous frame format.

How would I go about doing this? I have already trained the model on TuSimple for just CNN encoder.


texts:
I have a graph with vertices that represent some entities and the edges are weighted as the correlation between two such entities.

I would like to break this graph into several subgraphs with high inner-correlation.

My problem is that I have a few 'hubs', with high correlation to a lot of different entities.

How can I detect such hubs in order to remove them?


texts: Linux tools to handle large data sets (several TB)

texts:
I come from a physics background that has a very specific tool (ROOT, a C++ toolkit) to handle very large data sets for a very specific purpose.  For smaller data sets (MB to GB), I use modules in Python, R, or just good ole scripting with pipes for stdin/stdout, often with a relational database for storage.

My question is: what tools are available on Linux to handle terabyte+ sized data sets in other fields, such as data science, finance, web analytics, etc?  Are there any standard references on them, such as K&R for learning C?


texts: In a double Deep Q network what would happen if we switch the roles of both networks

texts:
We normally use the online network for action selection and the target network for evaluation , would there be a difference if we switched the roles?
Because in the case Of Double Q learning, we alternate between the two estimators, this isn’t the case in Double deep Q learning
If our target network is the one that actually overestimated it’s values how do we correct that
Eg
Q(s,a) =R + y Q(s’, argmax(Q*(s’,a,w-)),w)
In this example our online network has weights w
And our target network has weights w-
As you can see the target network performs action selection
And our online network performs evaluation
Would there be any issue with doing it this way?


texts: CUDA compatibility of GTX 1650ti versus 1650

texts:
I am confused about CUDA compatibility. I am studying deep learning and looking for a laptop to buy. One laptop has GTX 1650ti and another has GTX 1650. Will both be able to use GPU for model training, or only second one?
I checked for gpu compatibility. On the nvidia website only gtx 1650 is mentioned. But on some other forums I read that both can work.


texts: How to understand the weights and biases for beginners?

texts:
I am newbie to deep learning, I was building my first model using MNIST dataset, I understood the full  model, but one thing is a bit confusing to me. How can we get the weights and bias? Is it that, weights are our input data or is it going to get assigned some random values and also bias or some of the weights.

It works based on this formula

Z=Wh∗x+bh


texts: Perform bootstrapping of an ordinary linear regression model, using B=100 bootstrap resamples of my dataset, and getting RMSE

texts: So Im studying machine learning through R, and Im working with the boston data set from the library MASS. I am practicing bootsrapping. I already carried out analysis to determine how ,many  distinct data points on average are drawn from the sample to make up a bootsratp resample, using B=100 resamples of the dataset. Next I would like to do two things- perform boostrapping of an ordinary linear regression model using B=100 resamples of the data set again and use the OOB samples to predict RMSE. And aslo use bootstrapping of a ridge regression model with 100 Bootstrap samples to predict RMSE and then compare my answers. I am having difficulty adapting the code from what I have already done into this. Does anyone have any idea?


texts: "Each agent was evaluated every 250,000 training frames for 135,000 validation frames" What does this sentences stands for? in DQN nature paper?

texts:
In nature paper of DQN by DeepMind, DQN is compared to linear function but they does not said what is this linear function? They compared with some linear functions?

0- What is the meaning of this sentences "Each agent was evaluated every 250,000 training frames for 135,000 validation frames" in DQN nature paper?

2- What about meaning of this sentences:  "Note that these evaluation episodes were not truncated at 5min leading to higher scores on Enduro"

1- Does this correct:we just put a network with an input and an output?
I do not sense why the performance of it is very how?

TheperformanceoftheDQNagentiscomparedwiththeperformanceofalinearfunctionapproximator on the 5 validation games (that is, where a single linear layer was used instead of the convolutional network, in combination with replay and separate target network). Agents were trained for 10 million frames using standard hyper parameters,and three different learning rates. Each agent was evaluated every 250,000 training frames for 135,000 validation frames and the
 highest average episode score is reported. Note that these evaluation episodes were not truncated at 5 min leading to higher scores on Enduro than the one s reported in Extended Data Table2. Note  so that the number of training frames was shorter (10 million frames) as compared to the main results presented in Extended Data Table 2 (50 million frames).


texts: xgboost performance with predicted values as input

texts:
I have predicted the probability of loss using different features. Now when I used this with a non-important feature to predict the probability of loss. The first one is very close. logloss was close to 0.11. However, I have few more other features, I wanted to know if the features are important or not. So, I used the new features with this predicted probability. I found volatile behavior. Not only did the performance (logloss) drop to 0.14, but the model didn't pick the predicted probability as an important feature.

My Primary Questions:

What is the reason behind this outcome?
Should I dump every single
feature into one model, then see which feature is important ?


texts:
Hi all: I have a very fundamental question on how CNN works.

I understand fully the training process as to take a bunch of images, start with random filters, convolve, activate, calculate loss, back propagate and learn weights. Fully understood.

But once the training is done, the last convolution layer has the most complex and complete features like faces, ears, wheels and such filters that can get activated by full features.

If that is so, during testing, do we need to pass our images through all the layers again? Why don't we convolve our images against the last convolution layer and see how many of these complex feature filters get activated? And pass that on to the fully connected layer for classification?

I understand there might me inconsistencies in the layers and the inputs but except that anything more important?


texts: OOB decision function doesn't match prediction in scikit-learn RandomForest

texts:
I am using a RandomForest for multiclass classification. I would like to use the oob_decision_function to explore precision/recall, but I don't understand the OOB results.

I am using 25,000 trees (n_estimators=25000) and the possible class values are 0,1 or 2.

My results are (truncated to the first 10 rows):

y_train = [1 2 1 1 0 0 2 2 2 2]
y_pred  = [2 2 2 1 0 2 2 2 2 2]

oob_decision_function:
[[ 0.25377529  0.28080796  0.46541674]
 [ 0.32162915  0.3250808   0.35329005]
 [ 0.34463485  0.27709584  0.3782693 ]
 [ 0.31091392  0.2982096   0.39087648]
 [ 0.34932553  0.28632762  0.36434685]
 [ 0.31535905  0.19570567  0.48893528]
 [ 0.25472683  0.35845451  0.38681866]
 [ 0.32521156  0.31721116  0.35757728]
 [ 0.30706625  0.32703203  0.36590172]
 [ 0.29785305  0.22490485  0.4772421 ]]

The dataset is not uniform:

class 0:  32% of samples
class 1:  27% of samples
class 2:  41% of samples

The predictions don't seem to agree with the decision function. See for example the 4th sample: the prediction (y_pred[3]) is class 1, but the OOB values are 0.3109 (class 0), 0.2982 (class 1) and 0.3908 (class 2). Why is the prediction class 1 and not class 2? I thought after 25,000 trees the prediction should closely match the OOB probabilities, or am I not understanding OOB properly?

A few questions:

how are the OOB values calculated? I thought it was as follows: for
each of the 25,000 trees it creates a new bagged training set. Let's
say a given sample was not included (ie. out-of-bag) in 6000 of those
trees. When the sample is out-of-bag, the current tree is used to predict the
class for that sample. So for our given sample, it was estimated 6000
times and let's say the predictions were class 0 (1000 times), class
1 (2000 times) and class 2 (3000 times). Therefore the OOB values
would be 1000/6000 (class 0), 2000/6000 (class 1) and 3000/6000
(class 2). Is this correct?
the OOB values are calculated using the results of the predictions
from individual trees (as explained above), but the output
predictions (ie. y_pred) are from the final ensemble of trees. Is
this correct?
should the OOB values converge to the cross validation values as the
number of trees increases? AFAIK k-fold cross validation splits the
data into k subsets and uses the entire forest on each subset.
is there some sort of weighting being applied (e.g. if my data are
not uniformly distributed)?


texts: Classification followed by regression to handle response variable that is usually zero

texts:
I have a data set consisting of a bunch of predictors (mostly unbounded or positive real numbers) and a single response variable that I wish to predict. The response is typically exactly zero -- around 90% of the time. I have tried modelling this using standard Gaussian process methods as well as random forests. However, in both cases (although moreso when using random forests) the model seems to handle the data poorly, usually predicting a non-zero response. Now, if the predicted responses were in fact very close to zero I could just set a cut-off below which the values would be rounded to zero, but they are significantly non-zero in many cases.

My idea for a solution is to train two models: a classification model trained on the entire training-set that predicts whether a variable is zero or non-zero, and a regression model trained only on the rows in the training set with a non-zero response. I would then first use the classification model to predict which observations have a response that is exactly zero, and subsequently use the regression model to predict the value of the non-zero responses.

Is this a sound way to solve the described problem? Does this sort of model have a name? Are there better ways to do this?


texts:
I am working with GPS track files (list of X and Y coordinates). I have tracks with a high sampling rate and want to downsample the track for easier handling.

The obvious way would be to create a new list of points, and to keep only (for example) every 100th point of the track. The problem is that this could remove important extremes, such as curves.

Do you know algorithms, which allow to downsample the two dimensional array, while keeping the extremes (and dropping datapoints without much value, such as points which are in a straight line).

It would be great if there would be a solution in python, but I'm gratefule for all kind of help.

Thanks!


texts: How to export one cell of a jupyter notebook?

texts:
I'm currently working/prototyping into a Jupyter notebook. I want to run some of my code on a standalone iPython shell.

For now, I export my iPython code (file --> download as) and then execute it in my iPython (with %run). It works, but I would like to export only one cell or set of cells. So, that I can run only what I modified in my Jupyter notebook.


texts: Are there any tools for feature engineering?

texts: Specifically what I am looking for are tools with some functionality, which is specific to feature engineering. I would like to be able to easily smooth, visualize, fill gaps, etc. Something similar to MS Excel, but that has R as the underlying language instead of VB.


texts: Should I make single machine learning model for predicting price of house or 6 different models, given 6 datasets for different cities?

texts:
I am currently working on "Housing Prices in Metropolitan Areas of India" (kaggle dataset), with 6 different csv files for 6 different cities.
All have same columns (40).
Working on a real estate website project, I am confused, should I make separate model for each csv or merge all csv to form one model (after data cleaning) to predict house prices.


texts: Which classification algorithms to try for classifying text data into 300 categories

texts:
I have 40000 rows of text data of health care domain. Data has one column for text (2-5 sentences) and one column for its category.
I want to classify that into 300 categories. Some categories are independent while some are somewhat related. Distribution of data among categories is not uniform either i.e some of the categories(around 40 of them) have less data about 2-3 rows.

I am attaching log probablity of each class/categories. (OR distribution of classes) here.


texts:
I am currently trying to optimise some parameters on my model (15000 samples). What I am finding is a relatively large variance in the loss function 2%-10% which makes it hard to identify which parameter is the best. This appears to happen based on how the random number generator splits the data into train/test sets.

I have tried :

CV 5-fold
Split of 75%

Fixing the random seed does help (or using the same test set), but it concerns me that I get such variations based on what samples are in the test set. It seems alarming that the 'best parameter' is so dependent on a particular shuffle of the data and I worry how it translates to real world use.

What are people's approach to situations like this? I was thinking I could just repeat each test multiple times and take the average, but that has very large computation costs and seems very inefficient.


texts:
I have found a number of parsers for the automatic extraction of institution names from texts (e.g. this one). My task is in a sense the inverse one: I want to automatically generate reality-like institution names, with a possibility to differentiate them by type (private-held, public, educational, etc.) and by branch.
Is there any algorithm / application / papers to be found? Alternatively, is there any (free access) database with such data?


texts: Compute Jaccard distance between two lists of strings

texts:
I have input like below:

input_list = [['Search' 'engines','using','machine','learning','pattern','detections'],

        ['machine','learning','helped','Google','automatically','sift','pages']]

input_list1 = ['Machine','learning','ever','evolving','technology']

Expecting Jaccard similarity distance between input_list and input_list1.


texts: Getting transportation turnover as null when i filtered by Product nature

texts:
I am pretty new to SQL. When i tried the following query i am getting sum as 0.

Can somebody explain me the problem of the query.

select sum(transportation_turnover)
from(
SELECT  top 10000 dd.tdt_order_number AS order_number,
              dd.tdt_date_to_ordered AS ordered_date,
              dd.frm_delivery_type AS delivery_type,
              dd.dyh_date_expected_delivery AS promised_delivery_date,
              dd.dyh_date_actual_shipment AS warehouse_pickup_time,
              dd.ctm_zip_code AS customer_zip_code,
              dd.tdt_date_to_returned AS date_of_return,
              ds.family_label as family,
              ds.dpt_label as department,
              ds.product_nature_label as product_nature,
              ds.mdl_num_model_r3 AS model_number,
              SUM(CASE WHEN dd.tdt_item_id = '2102120' THEN dd.f_to_tax_in ELSE 0 END) AS transportation_turnover,
              SUM(dd.f_qty_item) as quantity,
              dd.tdt_type_detail as type,
              ds.mdl_label as model_name,
              but.but_name_business_unit as shipped_from
         FROM cds.f_delivery_detail_current dd
         INNER JOIN cds.d_sku ds ON dd.sku_idr_sku = ds.sku_idr_sku
         INNER JOIN cds.d_business_unit but ON dd.but_idr_business_unit_stock_origin= but.but_idr_business_unit
       WHERE dd.cnt_idr_country = 50
       AND   EXTRACT(YEAR FROM dd.the_date_transaction) IN (2019,2020)
       GROUP BY order_number,
                ordered_date,
                delivery_type,
                promised_delivery_date,
                warehouse_pickup_time,
                customer_zip_code,
                date_of_return,
                family,
                department,
                product_nature,
                model_number,
                type,
                model_name,
                shipped_from
       ORDER BY order_number)
where type = 'sale'


texts: Machine Learning - Where is the difference between one-class, binary-class and multinominal-class classification?

texts:
Where is the difference between one-class, binary-class and multinominal-class classification?

If I like to classify text in lets say four classes and also want the system to be able to tell me that none of these classes matches the unknown/untrained test-data.

Couldn't I just use all the methods that I mentioned above to reach my goal?
e.g. I could describe C1, C2, C3 and C4 as four different trainings-sets for binary-classification and use the trained models to label an unknow data-set ...

Just by saying, Training-Set for C1 contains class 1 (all good samples for C1) and class 0 (mix of all C2, C3 and C4 as bad samples for C1).

Is unlabeled-data C1 -> 1 or 0

Is unlabeled-data C2 -> 1 or 0
... and so on ...

For multinominal classification I could just define a training-set containing all good sample data for C1, C2, C3 and C4 in one training-set and then use the one resulting model for classification ...

But where is the difference between this two methods? (except of that I have to use different algorithms)

And how would I define a training-set for the described problem of categorizing data in those four classes using one-class classfication (is that even possible)?

Excuse me if I'm completely wrong in my thinking. Would appreciate an answer that makes the methodology a little bit clearer to me =)


texts: feature weights in structured support vector machine

texts: I like to find the weight vector for input-space features in a structured SVM. The idea is to identify the most important set of input-space features (based on the magnitude of their corresponding weights). I know that in a binary SVM the weight vector can be written as a linear combination of examples, and the magnitude of those weights represents how much they were effective for the prediction problem at hand. But how do you compute the same for an SSVM?


texts: Is there a quick way to speed up ICP in python using a cached KD-tree

texts:
I am currently using ICP to match 2 point clouds. These point clouds evolve in time, so I have to repeat this process many times. I am using a standard KD tree from scipy for my nearest neighbor lookup. Each point cloud has roughly 500 points each. I am looking at using a cached KD tree but I am unsure about how to turn a KD tree into a cached KD tree. Is there an implementation in Scipy? SVD in python seems fast enough for my needs at the moment, but I would also be open to any known faster SVD solvers. My current code for the ICP is:

>tree=spatial.KDTree(TimeStep1)
>qHat=TimeStep2
> for i in range(10):
>         Phat=tree.query(qHat,eps=.15,distance_upper_bound=2)
>         H = np.dot(qHat.T, TimeStep1[Phat[1]-1,:])
>         U, S, Vt = np.linalg.svd(H)
>         R = np.dot(Vt.T, U.T)
>         if np.linalg.det(R) < 0:
>             Vt[2,:] *= -1
>             R = np.dot(Vt.T, U.T)
>             Yes.append(1)
>         Eu=rotationMatrixToEulerAngles(R)
>         T=TimeStep1[Phat[1]-1,:].T-np.dot(R,qHat.T)
>         T=TimeStep1[Phat[1]-1,:]-qHat
>         T=T.mean(axis=0)
>         qHat=np.dot(R,TimeStep2.T).T+T
>         qHat=TimeStep2+T.shape


texts: transformers require a large d_model even when the input cardinality is low?

texts:
I'm training a transformer encoder for an NLP task over character data, so the cardinality of my input is 26. I've noticed that if I want to create a strong model, I need make $x$ == my embedding dimension == model dimension == output dimension larger, because the number of trainable parameters in my model heavily depends on $x$. Does it makes sense to represent characters with a large number of dimensions, e.g. $x=300$? In LSTM the number of parameters depends on the hidden size $h$ so I didn't have to do that...
Transformer encoder, single layer:

attention: $4\times[x\times x+x]$
norm: $2\times[x+x]$
linear: $2\times[x\times h]+x+h$

LSTM, single layer:

$4\times[(x+h)\times h+h]$


texts: Find average sequence from a set of sequences

texts:
I have a set of user sessions. Session consists of an ordered list of types of actions that user made (for example, bought a gun, played a mission, etc).
I want to create/calculate session that have most possible similarity to all provided sessions (most common types of actions users make in order they make them)

Unfortunately, I know nothing about data science but I tried to google a way to do that.
I've found this document: https://cran.r-project.org/web/packages/TraMineR/vignettes/TraMineR-state-sequence.pdf
And it looks like 9.1 and 9.2 describe things similar to what I want. But I dont know this for certain and even if it's true I stil don't know how to use it for my scenario.


texts: Modelling on one Population and Evaluating on another Population

texts:
I am currently on a project that will build a model (train and test) on Client-side Web data, but evaluate this model on Sever-side Web data.  Unfortunately building the model on Server-side data is not an option, nor is it an option to evaluate this model on Client-side data.

This model will be based on metrics collected on specific visitors.  This is a real time system that will be calculating a likelihood based on metrics collected while visitors browse the website.

I am looking for approaches to ensure the highest possible accuracy on the model evaluation.

So far I have the following ideas,

Clean the Server-side data by removing webpages that are never seen Client-side.
Collect additional data Server-side data to make the Server-side data more closely resemble Client-side data.
Collect data on the Client and send this data to the Server.  This is possible and may be the best solution, but is currently undesirable.
Build one or more models that estimate Client-side Visitor metrics from Server-side Visitor metrics and use these estimates in the Likelihood model.

Any other thoughts on evaluating over one Population while training (and testing) on another Population?


texts: Convert Pandas Dataframe with mixed datatypes to LibSVM format

texts:
I have a pandas data frame with about Million rows and 3 columns.  The columns are of 3 different datatypes. NumberOfFollowers is of a numerical datatype, UserName is of a categorical data type, Embeddings is of categorical-set type.

df:

Index  NumberOfFollowers                  UserName                    Embeddings        Target Variable

0        15                                name1                      [0.5 0.3 0.2]       0
1        4                                 name2                      [0.4 0.2 0.4]       1
2        8                                 name3                      [0.5 0.5 0.0]       0
3        10                                name1                      [0.1 0.0 0.9]       0
...      ...                               ....                       ...                 ..

I would like to convert this pandas data frame into the LibSVM input format.

Desired Output:

0 0:15 4:1 1:0.5 2:0.3 3:0.2
1 0:4 5:1 1:0.4 2:0.2 3:0.4
0 0:8 6:1 1:0.5 2:0.5 3:0.0
0 0:10 4:1 1:0.1 2:0.0 3:0.9
...

One solution I found was using:

https://scikit-learn.org/stable/modules/generated/sklearn.datasets.dump_svmlight_file.html

It takes input as a NumPy array or a sparse matrix.

The UserName has a million unique values, so calling pd.get_dummies on this column and storing it as a dense NumPy array is not a solution and will not fit in the memory.

So, I know it may be done using sparse matrices, but, then I don't know how to convert the above data with mixed data types into a sparse matrix and then use sklearn.datasets.dump_svmlight_file.html.

In reality, I have many columns with mixed data types and I need to convert them into libSVM format. But, all the columns fall in one of the above three types.

Thanks in advance for any thoughts on how to solve the above problem.


texts: PCA ? after the transformed data, are they still same with original data, (if maintain same dimensional)

texts:
after the step of pca, I try to plot them (see pic)

example in the original (A) 2 features => pca 2 features
if I do not reduce any dimensional data, are they still have the same meaning of these data? (the picture look a bit different from them)
it is a circle example for classification data.
any seem similar example after the step of pca.


texts: I have a two class classification problem and my neural network prior to training predicts with an accuracy of 80%. After training i have an accuracy of 75%. Can you tell me how this is possible?


texts: For imbalanced classification, should the validation dataset be balanced?

texts:
I am building a binary classification model for imbalanced data (e.g., 90% Pos class vs 10% Neg Class).

I already balanced my training dataset to reflect a a 50/50 class split, while my holdout (training dataset) was kept similar to the original data distribution (i.e., 90% vs 10%).
My question is regarding the validation data used during the CV hyperparameter process. During each iteration fold should:

1) Both the training and test folds be balanced

or

2) The training fold should be kept balanced while the validation fold should be made imbalanced to reflect the original data distribution and holdout dataset.

I am currently using the 1st option to tune my model; however, is this approach valid given that the holdout and validation datasets have different distributions?


texts:
I have three factors,

Income(from source A)，continuous variable
Income(from source B), continuous variable
Happiness index，continuous variable

Suppose I have 500 samples. My goal is to show the influence of both Income(from source A) and Income(from source B) on Happiness index. Meanwhile, I want to show the distribution of Happiness index. I think a boxplot on Happiness index is a good choice. I can use color to represent Income(from source A), however, how can I represent another continuous variable, i.e, Income(from source B) here?


texts: What is the structure and dimension of input passed to neural network when training CBOW and SKIP GRAM word embedding

texts:
I am confused about input passed to neural network in natural language processing (NLP) when training CBOW word embedding from scratch.  I read the paper and have some doubts.
In general neural network (NN) architecture, it is more clear that each row act's as input to neural network with d features. For example in the figure below:

X1, X2, X3 is one input, or one row of the data-frame. So here, one data point is of dimension 3 and data-frame would be like this:
X1  X2  X3
1   2   3
4   5   6
7   8   9

Is my understanding correct?
Now coming to NLP, CBOW architecture: Lets take an example to train CBOW word embeddings:
Sentence1: "I like natural processing domain."
Creating training data from above sentence, window size=1
Input                      output

(I,natural)                like
(like,processing)          natural
(natural,domain)           processing
(processing)               domain

Is the above creation of training data for CBOW architecture for window size=1 correct?
My Questions are below:
How will I pass this training data to neural network for the above figure?
If I represent every word as one-hot encoded vector of dimension equal to size of vocabulary V as input to neural network, then how should I pass 2 words at the same time of dimesion 2V as input.
Is this the way to pass the input for first training sample: I just concatanated the two input words:

Then I train the network to learn word-embeddings using cross entropy loss?
Is this the right way to pass input?
Secondly, the middle layer will give us the word embeddings for 2 input words or the target words??


texts: How to increase model's prediction accuracy

texts:
I am using the InceptionV3 model for training. Here is the link for the code (https://github.com/maxmelnick/tensorflow/blob/no_random/tensorflow/examples/image_retraining/retrain.py) Initially I have a small size dataset. So, I used the augmentation technique to increase the size of the dataset.
While training phase dataset was divided into training, validation, and testing. During the training phase, it shows 96% accuracy for 11 classes. But When I predict any new input image(Unseen data) it gave 56% accuracy. What's the problem lies with the model?
I have already used Dropout, Cross-validation, OverSampling techniques but not achieved good results over the new input image.
Parameters used while training.
Training Samples - 800 images in each class

Training Samples - 70%
Validation Samples - 20%
Testing Samples - 10%

Testing Samples (Unseen data other than Training Samples) - 51 images in each class
Epochs - 10,000


texts: class imbalance - applied SMOTE - next steps

texts:
I am new to ML and learnt a lot from your valuable posts. I need your advise with the following situation and guidance on if the steps make sense.
I have a binary classification problem, my dataset has a severe imbalance approximately 2% positive cases (4,000 cases) out of a total of 200,000 cases. I separated my dataset into a train and a test (80/20 stratified split). My train now has total of 160,000 cases (3,200 positive cases) and test has total of 40,000 (800 positive cases).

Next, from Train I created a (50-50) SMOTE sample which has ~9,000 positive cases (original 3,200 and SMOTE create synthetic samples of ~ 5,800 positives) and ~ 9,000 negative cases (so a total of 18,000 cases in the "new train").

next, I have developed bagged/ boosted tree based classifiers on the "new train" with ROC training criteria in R caret (I get high ROC around 0.9 which is usual). I then applied the model/s on the test set which has the original class imbalance and got the predicted probabilities (the AUC is now 0.65). I identified the optimal threshold for classification based on the pROC package for creation of the final class predictions on test dataset.

I need your advise on if I am transitioning from the trained SMOTE model with a 50/50 imbalance to the test model with the original imbalance of 50/1 correctly. is there anything fundamentally wrong in what I am doing and any suggestions on how to improve the process will be very helpful. are any corrections needed to the SMOTE based probabilities on test or is thresholding the best I can do?


texts: How far into the future can I forecast a time-series with an LSTM and strongly seasonal data

texts:
I am working on a Sequence-to-Sequence + Attention model for some time-series data. Now I have a really long time series, basically 40 years of daily observations for multiple sensors. The data itself comes from environmental sensors, so there is very distinctive seasonal trend. Here is a picture below. Right now I have a window of 365 days in and then predicting 365 days out. I could changes this to take in 730 days and predict out 365, and I would still have a lot of windows.

My question was, how long can I generally extend out a Sequence-to-Sequence model? Given the fact there is such strong seasonality in this data, seems like the long term behavior is very predictable, within some range of variation. If anyone has any references on how long an LSTM or Recurrent net can be extended, I would appreciate it.
Indeed, I did try and model this data with some ARIMA and SARIMA type models, but because most of the year the sensors are zero and then in the winter is where all the action is, the classical time-series models have a hard time determining the coefficients for the lags.


texts: How to fill missing enteries in column A, and add respective corresponding enteries to column-B with value of previous cell

texts:
I am facing an issue with an excel file. I have an excel sheet with 2 columns

Column A :  Time Increment with per second

Column B :  A particular value of a machine sensor

The problem i am facing is when the machine is stopped (not in motion), the depth increment stop for that particular time and do not make entries in the excel sheet, and once the machine start moving it again add the entries for the starting point. Let me explain in better way as below.

04:23:12    2709.082597
04:23:13    2708.747333
04:23:14    2708.442548
04:23:15    2708.229198
**04:23:16  2708.229198
04:24:01    2708.137763**
04:24:02    2707.924413

The machine is stopped at 04:23:16  and it resumed its activity at 04:24:01 with a gap of 44 seconds.So the excel sheet missed the time entries for those 44 seconds. There are more than 1000 entries like this in the file. What i want that Column A (Time) should be incremented from Cell A  to Cell Z, without missing any entry of the second (even when the machine is stopped), and corresponding value at column B should be added to all 44 cells which should be the value of sensor corresponding to the time 04:23:16 (when the machine stopped) i.e 2708.229198.

Like for example the machine is stopped at 04:23:16, Sensor value at Column B is 2708.229198, i want that file should fill all the missing entries for the time between 04:23:16 to 04:24:01 (44 seconds), and for all those 44 seconds the value at Column B should be 2708.229198 (A value of column B corresponding to 04:23:16), and this can be done for the whole sheet. As i explained its a very lengthy file and to do the option manually is not suitable, that may take hours or even days, I want some practical solution to sort out this issue.

Any help or guidance will be appreciated in this regard.

Jawad


texts: How to penalize for empty fields in a DataFrame?

texts:
I have to calculate the consistency of racing car drivers during the whole season. My DataFrame consists of 10 columns  (10 circuit names) and for each of those columns I have the standard deviation in lap time the driver posted in that circuit. In other words, how consistent the driver is from lap to lap. In races the driver did not finish the field is blank.

So far I have calculated their average season consistency by averaging all 10 columns. However, not finishing a race should affect a driver's consistency negatively and I do not know how to implement that.


texts: (Python Basic) more elegant way of creating a dictionary

texts:
Is there a more elegant way to write a code like this?

my_dic = {
    'Model':['Apple', 'Banana', 'Pineapple', 'Melon', 'Orange', 'Grape'],
    'AAA':[
        method1(y, y_Apple),
        method1(y, y_Banana),
        method1(y, y_Pineapple),
        method1(y, y_Melon),
        method1(y, y_Orange),
        method1(y, y_Grape)]
    ,'BBB':[
        method2(y, y_Apple),
        method2(y, y_Banana),
        method2(y, y_Pineapple),
        method2(y, y_Melon),
        method2(y, y_Orange),
        method2(y, y_Grape)]
    ,'CCC':[
        method3(y,y_Apple),
        method3(y, y_Banana),
        method3(y, y_Pineapple),
        method3(y, y_Melon),
        method3(y, y_Orange),
        method3(y, y_Grape)]
    ,'DDD':[
        method4(y, y_Apple),
        method4(y, y_Banana),
        method4(y, y_Pineapple),
        method4(y, y_Melon),
        method4(y, y_Orange),
        method4(y, y_Grape)]
    ,'EEE':[
        method5(y, y_Apple),
        method5(y, y_Banana),
        method5(y, y_Pineapple),
        method5(y, y_Melon),
        method5(y, y_Orange),
        method5(y, y_Grape)]
}


texts: Fine tune gpt2 via huggingface API for domain specific LM

texts:
i am using the script in the examples folder to fine-tune the LM for a bot meant to deal with insurance related queries. So if someone were to type "i am looking to modify my ..." , the autocomplete suggestions would be " modify my name ", "modify my surname", modify my vehicle number etc

My training data set has plenty of such samples BUT is always prefixed or suffixed by policy details, personal details etc which seems to be throwing off the fine-tuning and in predictions it always includes some random names , numbers /text , like "i want to modify my father's name into 1235..." Etc ..i hope u get the idea

One way of dealing with this issue would be to clean up the training dataset using some NER and get rid of specific information (not very impressive) or maybe unfreeze some other layers of the gpt2 model.

Tried exploring the API but it simply allows loading pre trained wirghts and explicitly unfreezes only the last classification layer (?) ..any pointers on resolving this would be extremely helpful.. appreciate any inputs


texts:
I am a little confused with using encog to create a neural network.

I am trying text classification with a basic feed forward network. For the input data I have 200 unique words (features/inputs) to input into the network and 100 different pieces of text. so I have a matrix of 100x230, essentially 100 different training items each with the frequency of 230 words. I then have an output which classifies the data as A, B, C, D or E, so one column and 5 different outputs, therefore this is an ordinal column and is part of the CSV data set I read in thus I actually have a matrix of 100X231 where the 231 column is the output I desire to classify.

I can train and run this neural network no problem but I am confused about the number of output neurons to have. I would have thought there are 5 different classifications therefore I should have 5 different neurons, however I can't setup the code like that, as it complains:

IMLDataSet trainingSet = new BasicMLDataSet(input, ideal);

My input variable is a double[100][230] and the ideal is just the column of expected classifications so double[100][1]. Because I only have one column and the data is normalised those A,B,C... values are turned into numbers between -1 and 1 and therefore when I run the neural net with 5 output neruons it complains that it should be only 1.

When I run the neural net with one neuron output it gives me what looks like correct and accurate answers but the value it gives is between -1 and 1. I am assuming the activation level is being matched to the output?

My question therefore is either how to denormalise this output to get back to my actual letter classification or how to use 5 neurons on the output so the network chooses the appropriate classification.

My neural network is as follows:

 BasicNetwork network = new BasicNetwork();
 network.AddLayer(new BasicLayer(null, true, 230));
 network.AddLayer(new BasicLayer(new ActivationTANH(), true, 16));
 network.AddLayer(new BasicLayer(new ActivationTANH(), false, 1));
 network.Structure.FinalizeStructure();
 network.Reset();


texts:
This is really weird, I have a real simple test dataset and built a really simple linear model on it:

smallData = data.head(1000)
# smallData = data

y = smallData['points'].values
X = smallData.drop(['points','country', 'description'], axis=1)

X = pd.get_dummies(X, columns=['designation', 'province', 'region_1', 'region_2', 'variety', 'winery', 'taster_name', 'taster_twitter_handle', 'title'])

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

regressor = LinearRegression()
regressor.fit(X_train, y_train)

y_pred = regressor.predict(X_test)

It actually works really fine, as long as I only train it with ~200 rows of data. Once I have 300 rows and more, it returns COMPLETELY wrong predictions for some rows. Like, my scale is 80 to 100, and the prediction says it should be five million. That seems slightly off, especially there is no value of five million in the training data, 100 is the highest.

My data has quite a lot of columns due to the dummy encoding. It seems like it's overfitting?

What can I do to only get reasonable predictions between 80 and 100?


texts: Convert exponential to normal distribution

texts:
For the distribution shown below, I want to convert the exponential distribution to a normal distribution. I want to do this is as part of data pre-processing so that the classifier can better interpret the feature (named ipc here).

The regular log transformation does not work here because of the (x-axis) spread.

How can I transform this data to a normal distribution?

A related answer has been pointed out in the comment but I am looking for some Python code excerpt as well.

Thanks


texts: What is the way to modify a neural network classifier to deal with sample points from outside of the label set?

texts: I am solving an image classification problem. However some photos may not belong to any category, and I'd like not to give any fake information, rather to capture this situation. What are the ways to do it. One idea I have in mind is to give an extra category "OTHER" but then I need to have some data for this category which would be difficult to gather as it's too general.


texts: How to create a general plot for this problem

texts:
I would like to have some ideas about each customer depending on how many products he sells each month.
We have created a new feature X based on some other features.
We have noticed that, in most case, when Feature X decrease the amount of product sells decrease as it is illustrated with the figure bellow

For each customer, I have a similar plot (i.e in the most cases when the feature decrease buying product decrease).
How can I generalize this analysis for getting one global plot?


texts: Extracting specific portions of text from poorly formatted document?

texts:
I have a corpus of text files (free books) which are poorly formatted.  The goal is to extract a particular chapter (say chapter 2) from the raw text with all weird formatting removed.  Some documents are literal copy and pastes from pdfs, so that the text contents from a chapter would have page numbers interspersed throughout, etc., and I.e., it would be impossible to create a good regex rule.

I've manually gone through a number of documents and stripped out all unwanted formatting to arrive at the desired, clean section of text.  Using this as the training set, what would be a good model to set up for this task?

To clarify, the input would be a (very long) string, and the output would also be a string.

EDIT:

Please note that this particular example can be very easily handled by a series of regex rules, but it's only for illustration purposes only.  Imagine if each document had their own minor idiosyncrasies making it very difficult to come up with a general series of regex rules to apply to all docs in the corpus.

Sample input:

Book Title ABC

Table of Contents

Preface ................................ Pg 1
Chapter 1 .............................. Pg 2
Chapter 2 .............................. Pg 3
Chapter 3 .............................. Pg 4

Preface

<p> Contents for the preface sit here.  Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse scelerisque turpis ac commodo porttitor. Suspendisse iaculis lacinia ante, non suscipit purus venenatis et. </p>

- 1 -

Leftover Contents from Preface

Chapter 1

Praesent at justo sit amet nulla porttitor hendrerit. Etiam pretium sem ac feugiat posuere. Nulla mi risus, mollis sit amet ligula egestas, lacinia dapibus tellus. Proin iaculis ligula ultricies nunc elementum maximus. Nulla accumsan libero at nunc scelerisque, ut fermentum lacus euismod. Nulla imperdiet pharetra laoreet. Quisque id mollis diam. Vestibulum interdum orci at lectus ullamcorper pharetra. Curabitur laoreet mollis pharetra.

- 2 -

Chapter 2

Cras diam nibh, congue sit amet imperdiet quis, finibus imperdiet tellus. Sed imperdiet risus id elit consequat cursus. Curabitur consequat facilisis molestie. Ut varius urna vel ornare scelerisque. Nullam dictum tempus sapien, in pulvinar tellus consectetur non. Vestibulum consequat pretium iaculis.

- 3 -

Leftover contents from Chapter 2.  Vivamus pretium mauris at metus egestas, eget luctus ligula auctor.

Chapter 3

Maecenas vel dapibus lorem. Sed eget justo sit amet libero aliquam maximus a quis augue. Nunc consequat, urna quis elementum condimentum, quam arcu consectetur tellus, vehicula gravida est purus id nisl.

- 4 -

Nulla eget molestie velit, pharetra euismod magna. Vestibulum ultricies justo vitae massa vulputate, et venenatis orci fringilla.

Index

Sample Output:

Cras diam nibh, congue sit amet imperdiet quis, finibus imperdiet tellus. Sed imperdiet risus id elit consequat cursus. Curabitur consequat facilisis molestie. Ut varius urna vel ornare scelerisque. Nullam dictum tempus sapien, in pulvinar tellus consectetur non. Vestibulum consequat pretium iaculis.
Leftover contents from Chapter 2.  Vivamus pretium mauris at metus egestas, eget luctus ligula auctor.


texts: Price optimization for tiered and seasonal products

texts:
Assuming I can collect the demand of the purchase of a certain product that are of different market tiers. Example: Product A is low end goods. Product B is another low end goods. Product C and D are middle-tier goods and product E and F are high-tier goods.

We have collected data the last year on the following
1. Which time period (season - festive? non-festive?) does the different tier product reacts based on the price set? Reacts refer to how many % of the product is sold at certain price range
2. How fast the reaction from the market after marketing is done? Marketing is done on 10 June and the products are all sold by 18 June for festive season that slated to happened in July (took 8 days at that price to finish selling)

How can data science benefit in terms of recommending
1. If we should push the marketing earlier or later?
2. If we can higher or lower the price? (Based on demand and sealing rate?)

Am I understanding it right that data science can help a marketer in this aspect? Which direction should I be looking into if I am interested to learn about it.


texts: Gradient Descent Step for word2vec negative sampling

texts:
For word2vec with negative sampling, the cost function for a single word is the following according to word2vec:
$$
E = - log(\sigma(v_{w_{O}}^{'}.u_{w_{I}})) - \sum_{k=1}^K log(\sigma(-v_{w_{k}}^{'}.u_{w_{I}}))
$$

$v_{w_{O}}^{'}$ = hidden->output word vector of the output word

$u_{w_{I}}$ = input->hidden word vector of the output word

$v_{w_{k}}^{'}$ = hidden->output word vector of the negative sampled word

$\sigma$ is the sigmoid function

And taking the derivative with respect to $v_{w_{O}}^{'}.u_{w_{j}}$ is:

$
\frac{\partial E}{\partial v_{w_{j}}^{'}.u_{w_{I}}} = \sigma(v_{w_{j}}^{'}.u_{w_{I}}) * (\sigma(v_{w_{j}}^{'}.u_{w_{I}}) - 1)
$ $ if w_j = w_O $

$
\frac{\partial E}{\partial v_{w_{j}}^{'}.u_{w_{I}}} = \sigma(v_{w_{j}}^{'}.u_{w_{I}}) * \sigma(-v_{w_{j}}^{'}.u_{w_{I}})
$ $ if w_j = w_k \ for \ k = 1...K$

Then we can use chain rule to get

$ \frac{\partial E}{\partial v_{w_{j}}^{'}} = \frac{\partial E}{\partial v_{w_{j}}^{'}.u_{w_{I}}} * \frac{\partial v_{w_{j}}^{'}.u_{w_{I}}}{\partial v_{w_{j}}^{'}} $

Is my reasoning and derivative correct? I am still new to ML so any help would be great!


texts: Saving the Autoencoder predicted image to a new directory

texts:
I trained an autoencoder model in Keras to generate denoised images given noisy images. The predicted images are stored in the "result" directory, with the individual filenames appended with the "denoised" tag. When I run the code, nothing is stored in the "result" folder. When I ran for an individual image, the final cv2.imwrite throws this error: "TypeError: Expected Ptr<cv::UMat> for argument 'img'". Is there a different way to write the predicted files to the destination folder without using openCV in this case?
import glob
from keras.preprocessing import image

test = glob.glob("data/test1/*.png") #all test data that needs to be denoised
test.sort()
result = "data/result" #the folder that contains predicted denoised images
#load model
model = load_model('autoencoder_model.h5')
model.summary()
#compile the model
model.compile(loss='mean_squared_error', optimizer = RMSprop())

for f in test:
    img = image.load_img(f,grayscale=True,target_size=(128,128),interpolation="nearest")
    img_name = f.split(os.sep)[-1]
    #preprocess the image
    x = image.img_to_array(img)
    x = np.expand_dims(x, axis=0)
    x /= 255
    #predict on the image
    preds = model.predict(x)
    pred_norm = 255 * (preds - preds.min()) / (preds.max() - preds.min())
    pred_normed = np.array(pred_norm, np.int)
    cv2.imwrite(os.path.join(result,f[:-4]+'_removed.png'), pred_normed)


texts: Can Recurrent Neural Networks (LSTM) run distributed in production?

texts:
I have a question about RNN's based on LSTM cells. Currently i'm trying to predict anomalies in time series data, based on the prediction error. Is it reasonable to run RNN's distributed in production? For example to scale it with Google Cloud ML Engine.
I want to be able to scale the model in case it has to compute to many requests during inference.

But when I distribute the model what will happen to the memory cell? The data is split and distributed over multiple nodes, will it still recognize the pattern of the time series data?

Thnx!


texts: Cluster based on both positions and similarity scores

texts:
I have a dataframe position giving me the x and y positions of 87 points.
I also have a 87 x 87 similarity matrix giving me the pairwise similarity scores between those points.
My goal is to form groups of points taking into account both the spatial distances and the similarity scores between the points and to find relevant metrics to quantify the quality of the resulting clusters.

I do not have ground truth labels and I do not have prior beliefs on how many clusters I should end up with.
After a bit of investigation, I noticed that similarity is usually inversely proportional to distance: close points are more similar than far away points but it happens some times.
My similarity scores varies from 0 to 894. it is a very sparse matrix (1035 non zero values), oddly enough it has some zeros on the diagonal, the mean value is 4 and the std is 25. the max value out of the diagonal is 400.

So far I have tried:

to form clusters taking only the similarity scores into account using AffinyPropagation. This seems to give the most visually pleasant clusters so far but I still need a metric to quantify the quality. This gives me 15 clusters with some overlapping
to convert my positions into an 87x87 euclidean distances matrix and then this matrix into a spatial affinity matrix, and combine the spatial affinity and the similarity into a single affinity measure by adding them or multiplying them and then applying affinity based clustering algorithms.
to convert my similarity matrix into a distance matrix by defining d(x1,x2) = min_x3{ 1/sim(x1,x3) + 1/sim(x3, x2)} (I am not sure if the triangle inequality holds but it has been suggested here) and then combine my euclidean distances and these distances into a single distance by adding them or multiplying them and then applying distance based clustering algorithms.
to use spectral embedding such as laplacian eigenmaps using only the similarity scores to map my points into a 2d or 3d embedding and use clustering algorithms from there. This does not seem to work very well: I end up with 2 clusters and I think the right number should be around 10-15
I also tried all kind of transformation on the similarity matrix: making it symmetric (it is originally not symmetric), normalizing by dividing by the max value, capping to a percentile and then  normalizing, filling the diagonal with ones... it does not seem to make a big difference in the clusters

I am running out of ideas and looking for the general way of dealing with such problems if any exists and for metrics to be able to compare my different approaches.

As I do not know the ground truth labels, according to scikit learn documentation I can only use SilhouetteCoefficient, calinski harabasz index, and davies bouldin index but I am not very familiar with thesee. I also thought of Percentage of Variance Explained as used in the Elbow Method to find the best number of clusters.

Any suggestion would be greatly appreciated!


texts: Semi Supervised Learning without label propagation

texts:
I am trying to cluster some words by affinity.
Using Word2Vec I obtained vector representation of every word that I can cluster with a normal unsupervised method.

Of these words, though, I know the classification of some of them, for example I know:

Colors group ==> red, blue, yellow
Gender group ==> man, woman

UNKNOWN ==> shoes, brown, beautiful.

As the example shows, brown should be categorized as a Color, but shoes and beautiful should be different clusters.

How could I use this information to create a semi-supervised model in order to cluster every word?

While the question is generic, I actually tried to create a program in Python, I tried the scikit methods sklearn.semi_supervised.LabelSpreading and sklearn.semi_supervised.LabelSpreading.
These, though, are not what I need since they only assign the known labels to the remaining objects, meaning that I would end up only with my Colors and Gender group.


texts: non-linear optimization for a linear classifier? (scikit-learn)

texts: Using scikit-learn, why would you use bfgs optimization which is non-linear for a linear classifier as logistic regression? I am confused. Does the optimization method finds the optimum of the chosen score function? if so, which one? I can't choose it when defining the estimator. does the linearity or non-linearity of the score function depend on the model (whether it is linear or non-linear)?


texts: What are the parameter estimation methods for the two methods in this paper?

texts:
Automated Time Series Forecasting for Biosurveillance

in the above paper, page 4, two models, non-adaptive regression model, adaptive regression model, the non-adaptive regression model's parameter estimation method is "least squares", what is the parameter estimation for the adaptive regression model? is there any package in R to do parameter estimation for this kind of adaptive regression model? If I add more predictors in the adaptive regression model, can R still solve it? and how?


texts:
I've defined a function in this way:

def qfun(par):
    return(par[0]+atan(par[3])*par[1]+atan(par[4])*par[2])

How can I obtain the gradient of this function for only some of the elements (par [0:2]) in a specific point? I only find functions with only one "x", so for those cases it is simple, but when your function has more parameters what should I do?