texts: How do I arrange my data to predict 6 weeks of daily sales

texts:
I have a data.table base that has many variables to use them to forecasting sales for the next 6 weeks of daily sales. In fact, all the database is arranged by date as you can see here.Note that here I just show you some of variables.

> Data_train[order(Date)]
         Store DayOfWeek       Date Sales Customers Open Promo StateHoliday SchoolHoliday
      1:     1         2 2013-01-01     0         0    0     0            a             1
      2:     2         2 2013-01-01     0         0    0     0            a             1
      3:     3         2 2013-01-01     0         0    0     0            a             1
      4:     4         2 2013-01-01     0         0    0     0            a             1
      5:     5         2 2013-01-01     0         0    0     0            a             1
     ---
1017205:  1111         5 2015-07-31  5723       422    1     1            0             1
1017206:  1112         5 2015-07-31  9626       767    1     1            0             1
1017207:  1113         5 2015-07-31  7289       720    1     1            0             1
1017208:  1114         5 2015-07-31 27508      3745    1     1            0             1
1017209:  1115         5 2015-07-31  8680       538    1     1            0             1 .

My question is about the arrangement of the data according to the target goal. My problem is really about the Date variable.
In fact, I suggest this path:

I sum all sales by each date (because I have many types of stores).
I order my database according to the Date 's ascending order.
I didn't need duplicated rows in the date variables so I delete them.

Just to show you the new base for considered variables.

> Data_train[,SumSaleseachDay:=sum(Sales),by=c('Date')][order(Date)][!duplicated(Date)][,-c('Sales','Customers'),with=FALSE]
     Store DayOfWeek       Date Open Promo StateHoliday SchoolHoliday SumSaleseachDay
  1:     1         2 2013-01-01    0     0            a             1           97235
  2:     1         3 2013-01-02    1     0            0             1         6949829
  3:     1         4 2013-01-03    1     0            0             1         6347820
  4:     1         5 2013-01-04    1     0            0             1         6638954
  5:     1         6 2013-01-05    1     0            0             1         5951593
 ---
938:     1         1 2015-07-27    1     1            0             1        10707292
939:     1         2 2015-07-28    1     1            0             1         9115073
940:     1         3 2015-07-29    1     1            0             1         8499962
941:     1         4 2015-07-30    1     1            0             1         8798854
942:     1         5 2015-07-31    1     1            0             1        10109742

ADDED INFORMATION:
I have a database of 1017209 rows. And for each Store, I have its historic of  Sales between 2013-01-01 and 2015-07-31. And I have also 17 variables included to build the model.

The steps above just lead to forecast by day.

If I want to forecast for each Store and by day, what should I do?
thank you in advance!


texts:
My course textbook writes this about weight spaces when talking about linear classification:
.
Why would $w^{T}x$ be considered to be in the negative half space if 0 is not less than 0?


texts: Visualizing a graph with a million vertices

texts:
What is the best tool to use to visualize (draw the vertices and edges) a graph with 1000000 vertices? There are about 50000 edges in the graph. And I can compute the location of individual vertices and edges.

I am thinking about writing a program to generate a svg. Any other suggestions?


texts: Detect the important part of a receipt (CNN)

texts:
In general I want to detect prices and products from a receipt. My approach was to detect the important part of the receipt (products with their prices) to then afterwards pass to firebase ml kit to detect text (OCR).

I build my model already but I'm stuck on the preprocessing part here.

I figured stretching the images to a fixed size and grayscaling it or even make it only black and white. What I don't get since I'm resizing each picture, how do I have to change the coordinates (labels as seen from AI perspective) from the polygon since I'm resizing the picture? Furthermore how should I rotate the bill to be perfectly level?

Any suggestions or help really appreciated!


texts: If my model is overfitting the training dataset, does adding noise to training dataset help regularizing the machine learning model

texts: I would like to know if this is a best practice or not. Can we add noise to the training data to help the model "fit less the training data"; as a result, hoping to generalize better on new unseen data?


texts: Appending to numpy array for creating dataset

texts:
I want to create a dataset from three numpy matrices - train1 = (204,), train2 = (204,) and train3 = (204,). Basically all sets are of same length. I am applying a sliding window function on each of window 4. Each set become of shape =(201,4) I want a new array in which all these values are appended row wise. Like for first train1 then train2 then train3. And final output set is of size =(603,4).

This is a sliding window function which converts array of shape (204,) to (201,4)

def moving_window(x, length, step=1):
    streams = it.tee(x, length)
    return zip(*[it.islice(stream, i, None, step) for stream, i in zip(streams, it.count(step=step))])

Create dataset fucntion is:

def create_dataset(dataset1,dataset2):
    dataX=[]
    x=list(moving_window(dataset1,4))
    x=np.asarray(x)
    dataX.append(x)
    y=list(moving_window(dataset2,4))
    y=np.asarray(y)
    dataX.append(y)
    return np.array(dataX)

data_new=create_dataset(train1,train2)

It is returning a dataset of shape 0(2,201,4). I think this is appending differently, but I want row wise appending. so that the new _dataset is of shape= (402,4) with two sets and (603,4) with three sets. I want to generalize as well like if I want for 10 training sets or twenty training sets. How can I do that?


texts: Why does an SVM model store the support vectors, and not just the separating hyperplane?

texts:
In every explanation of SVMs, we're shown how training finds a hyperplane that best separates the data.  Presumably then for inference, you just check which side of the plane a point is on.

However, all the "disadvantages of SVMs" posts [1, 2] lament that SVM models are large and slow because they end up storing most of the data as support vectors.

Why would SVMs store any of the data, rather than just the (coefficients of the) separating hyperplane?  (And what is a "support vector" in the soft-margin case, when points of both classes are scattered on both sides of the hyperplane, anyway?)


texts: Why is each successive tree in GBM fit on the negative gradient of the loss function?

texts:
Page 359 of Elements Of Statistical Learning 2nd edition says the below.

Can someone explain the intuition & simplify it in layman terms?

Questions

What is the reason/intuition & math behind fitting each successive tree in GBM on the negative gradient of the loss function?
Is it done to make GBM more generalization on unseen test dataset? If so how does fitting on negative gradient achieve this generalization on test data?


texts: Is it always a good idea to remove features that have high mutual information with each other and to remove features that have very low mutual information with the target variable? Why or why not?


texts: What Base Should Be Used For Negative Log Likelihood?

texts: When calculating the negative log likelihood loss, what base of log are we supposed to use?


texts: Log file analysis: extracting information part from value part

texts:
I'm trying to build a data set on several log files of one of our products.

The different log files have their own layout and own content; I successfully grouped them together, only one step remaining...

Indeed, the log "messages" are the best information. I don't have the comprehensive list of all those messages, and it's a bad idea to hard code based on those because that list can change every day.

What I would like to do is to separate the indentification text from the value text (for example: "Loaded file XXX" becomes (identification: "Loaded file", value: "XXX")). Unfortunately, this example is simple, and in real world there are different layouts and sometimes multiple values.

I was thinking about using string kernels, but it is intended for clustering ... and cluseting is not applicable here (I don't know the number of different types of messages and eventhough, it would be too much).

Do you have any idea?

Thanks for your help.

P.S: For those who programs, this can be easier to understand. Let's say that the code contains as logs printf("blabla %s", "xxx") -> I would like to have "blabla" and "xxx" seperatated


texts: Stacked Area / Stack Bar Plot with color map based on range of values (Python, Pandas)

texts:
I'm trying to create a plot like the one shown in the picture below - where for each x value (a trading period) the y values (Megawatt, MW, see table below) are stacked on top of each other in order of the price (shown in the table below). I then need each megawatt segment to be colored in terms of the price, so that the general price at any given MW level can be easily read from the graph.
The table below shows example data for one X-value (one trading period). This is what I wish to be stacked.
I thought that this might be possible by using either a stacked bar plot or a stacked area graph and assigning a colormap, however most implementations of these I can find using Pandas, Matplotlib, or Seaborn require each level across the graph to be its own series, which is then assigned a color. This would be an issue for me as the number of values I wish to stack may change for each x-value, and I want the color of each segment (Megawatt) to reflect the price associated with that segment (DollarsPerMegawattHour).
Any suggestions as to how this may be achieved? Having a color scale legend as shown in the example below would also be good...


texts: CNN implementation low accuracy on MINST data

texts:
I'm trying to implement VGG11 (Model A of Table 1 from this article) on the MINST dataset but I'm getting ~10% train & test accuracy (as bad as random guessing). I had to resize the MINST data from 28x28 to 32x32 to fit the CNN architecture. This is what I did:

from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense, Conv2D, MaxPooling2D, Flatten
from keras import optimizers, utils
from PIL import Image, ImageFilter
import numpy as np
import tensorflow as tf

# Preprocessing

x_size = 6000 # Changed to reduce training time
(x_train, y_train), (x_test, y_test) = mnist.load_data()
x_train_ = np.ndarray((x_size, 32, 32))
x_test_ = np.ndarray((x_test.shape[0], 32, 32))

# Resizing inputs to 32x32
for i in [0, x_size-1]:
    im = Image.fromarray(x_train[i], mode=None)
    im = im.resize((32, 32))
    x_train_[i] = np.array(im)
for i in [0,x_test.shape[0]-1]:
    im = Image.fromarray(x_test[i], mode=None)
    im = im.resize((32, 32))
    x_test_[i] = np.array(im)

x_train_ = x_train_.reshape(x_train_.shape[0], 32, 32, 1)
x_test_ = x_test_.reshape(x_test_.shape[0], 32, 32, 1)

y_train = utils.to_categorical(y_train,10)
y_test = utils.to_categorical(y_test,10)
y_train_ = y_train[:x_size]

# Model A (VGG11) of Table 1: ConvNet configurations from paper arXiv:1409.1556v6

model = Sequential()
model.add(Conv2D(64, kernel_size=(3, 3), strides=(1, 1), activation='relu', padding='same', input_shape=(32, 32, 1), data_format='channels_last'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2,2)))
model.add(Conv2D(128, kernel_size=(3, 3), strides=(1, 1), activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2,2)))
model.add(Conv2D(256, kernel_size=(3, 3), strides=(1, 1), activation='relu', padding='same'))
model.add(Conv2D(256, kernel_size=(3, 3), strides=(1, 1), activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2,2)))
model.add(Conv2D(512, kernel_size=(3, 3), strides=(1, 1), activation='relu', padding='same'))
model.add(Conv2D(512, kernel_size=(3, 3), strides=(1, 1), activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2,2)))
model.add(Conv2D(512, kernel_size=(3, 3), strides=(1, 1), activation='relu', padding='same'))
model.add(Conv2D(512, kernel_size=(3, 3), strides=(1, 1), activation='relu', padding='same'))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2,2)))
model.add(Flatten())
model.add(Dense(4096, activation='relu'))
model.add(Dense(4096, activation='relu'))
model.add(Dense(1000, activation='relu'))
model.add(Dense(10, activation='softmax'))

# Model compilation

model.compile(loss='categorical_crossentropy', optimizer=optimizers.SGD(lr=0.001, momentum=0.9, nesterov=True, clipnorm=1.), metrics=['accuracy'])

# Model fitting

model.fit(x_train_, y_train_, epochs=1, batch_size=32)

# Model evaluation

score = model.evaluate(x_train_, y_train_)
print('Train loss after 1 epoch:', score[0])
print('Train accuracy after 1 epoch:', score[1])

I've tried normalizing the input, changing training sizes, increasing epochs, changing FC/filter size, and changing optimizers (and learning rate). Train accuracy is as low from both the evaluation report and TensorFlow's History report. I'm expecting >95% accuracy. What am I doing wrong?


texts: Optimal Dimension of Graph(Vertex) Embedding

texts:
Let's define a embedding of a graph structure G = (V,E) where $\mid V\mid=v, \mid E \mid=e$
Now define an embedding $f: V \to R^d$ where $d\in \Bbb N$, an optimal dimension of embedding which contains every edge information of $G$.
(G is a directed graph and there exist no weight, thus it's not a network.)
I'd like to find an infimum formula of $d$ represented with $v$ and $e$.

[Backgroud of this problem]
I am trying to construct a neural network which can discern whether the given explanation of word is about the word "be" or the word "exist".
For example "having a real existence" is "exist".

To do this, first I need to find the most smallest dimension of each words' corresponding embedding of vertex for training of my network.


texts: Is the neural network in DQN used to learn like a supervised model?


texts: How should ethics be applied in data science

texts:
There was a recent furore with facebook experimenting on their users to see if they could alter user's emotions and now okcupid.

Whilst I am not a professional data scientist I read about data science ethics from Cathy O'Neill's book 'Doing Data Science' and would like to know if this is something that professionals are taught at academic level (I would expect so) or something that is ignored or is lightly applied in the professional world. Particularly for those who ended up doing data science accidentally.

Whilst the linked article touched on data integrity, the book also discussed the moral ethics behind understanding the impact of the data models that are created and the impact of those models which can have adverse effects when used inappropriately (sometimes unwittingly) or when the models are inaccurate, again producing adverse results.

The article discusses a code of practice and mentions the Data Science Association's Code of conduct, is this something that is in use? Rule 7 is of particular interest (quoted from their website):

  (a) A person who consults with a data scientist about the possibility
  of forming a client-data scientist relationship with respect to a
  matter is a prospective client.

  (b) Even when no client-data scientist relationship ensues, a data
  scientist who has learned information from a prospective client shall
  not use or reveal that information.

  (c) A data scientist subject to paragraph (b) shall not provide
  professional data science services for a client with interests
  materially adverse to those of a prospective client in the same or a
  substantially related industry if the data scientist received
  information from the prospective client that could be significantly
  harmful to that person in the matter

Is this something that is practiced professionally? Many users blindly accept that we get some free service (mail, social network, image hosting, blog platform etc..) and agree to an EULA in order to have ads pushed at us.

Finally how is this regulated, I often read about users being up in arms when the terms of a service change but it seems that it requires some liberty organisation, class action or a senator to react to such things before something happens.

By the way I am not making any judgements here or saying that all data scientists behave like this, I'm interested in what is taught academically and practiced professionally.


texts: I always here about TensorFlow is good because it is used for deploying and production. Does that mean that people don't use Keras for deploying models? If keras is now integrated into TensorFlow, does that mean that it can also be used for deployment and production?


texts: Equations in "Intoduction to RL": What is the meaning and difference between E, and E with subscript?

texts:
This question is from An introduction to RL, page 78. In the formula below the page, both

  $\mathbb{E}$ and $\mathbb{E_\pi}$

are mentioned. Could you help me understand the difference between these two in this page and in general?


texts:
I am focusing on implement an automated solution that will take as input many time series data (one at a time).

After padding any missing values in order to have a time series without gaps, I am trying to check whether each one is stationary or not and then proceed into using either hybridModel() or auto.arima() in order to get a proper forecasting.

One of the questions that have arisen is that I haven't found a complete solution/algorithm describing the steps one has to take to do so.

Specifically it is not clear to me what to do if p-value is not < 0.05 and is for instance between 95% and 90% , i.e. 0.05 < p-value < 0.1. Is it stationary or not?
For instance after decomposing I get the screenshot below:

I wonder whether I have to remove the trend in the series, and if that is the case how I could do so. It is not clear to me after the decomposition whether I have to get for instance the log() of my time series or find a way to remove the trend. Can this task be automated or it needs to be "supervised" in terms of having to beforehand examine the series decomposition etc?
Why the seasonal seems so repeating even thought my time series shows one distinct spike during
Finally, I found information about the procedure online but I haven't found a tutorial of explanatory step-by-step guide in order not to replicate existing solutions but to be able to produce tailored solutions as there are huge differences between the time series I examine.

Sorry for the long post, I tried to gather some time series questions in one place that I think will benefit others as well.


texts: How does one interpret output from hyper-engine while optimizing a single hyperparameter of a neural network?

texts:
I am currently trying to optimize the learning rate of a neural network built in tensorflow. The network has 3 hidden layers, with 500, 250 and 100 neurons respectively.
I have adapted code from an example provided by the author of hyper-engine, a python package for hyperparameter optimization that can be used with tensorflow.
The example may be found here.
Essentially all that has been altered is the definition of the model provided for hyperparameter optimization. The range over which the learning rate will be evaluated has also been narrowed by an order of magnitude. The current search space is from 0.1 to 0.01.
The output from this python script is an xjson file. The xjson consists of multiple entries of a single key value pair, {'points' : 'values'}. 'values' quite clearly represents the accuracy of the classifier on the test data. The key, 'points' seems like it should represent the learning rate corresponding to the respective accuracy for that key value pair. However, the values taken on by points range from 0.000002 - 1.
For this reason, I'm finding quite difficult to interpret the meaning of 'points'. I also failed to find any information about interpretation of the output on the github for this package.
If anybody could help me understand the output from this code, it would be greatly appreciated.


texts:
I have one assignment that I have four files
1) train_data.csv: The training file contains two fields (text, id).
2) train_label.csv: The label file contains two fields (id, label).
3) test_data.csv: The test file contains two fields (text, id).
4) sample_submission.csv: This is a file that needs to be submitted.
And this should be obvious multilabel classification, but whenever I try to identify labels in train data, it doesn't show labels.
How can I remove noise from train_data??
Any type of suggestion welcome!!

train_label:

train_data:


texts: convert significant features to a set of rules or information

texts:
Is there any way to set up some rules from features in a classification model?

Assume that we want to classify an employee as someone who will be terminated or not. We found that average hourly pay and working hours are the two most significant features for the model. Is there any way that we can say how changes in these two features impact the chance?

For example, employees whose average pay is less than $20 per hour have a higher chance of quitting or those who work more than 40 hours per week have a higher chance?

Is there anyway to come up with these thresholds?


texts:
When processing texts written in a human-readable language, typically a first step is tokenization, where we have a dictionary of tokens (words) and we collapse each substring in that dictionary into a single token.  For a language like English, the dictionary is known in advance.

In my case, I want to parse files with a format not based on any human language.  I don't have a dictionary of known tokens; instead, I want to learn this dictionary.  I do have lots of sample data I can analyze.

Are there techniques for generating a dictionary, given many example files?

I expect that any common substring that appears in many files is a good candidate for a token, so an obvious approach is to look for all common substrings and add them to the dictionary.  However, a complication is that there is no equivalent of whitespace that separates tokens, so it seems like some filtering might be needed to avoid treating phrases (sequences of multiple tokens) as a new token.  For example, imagine if we had a large corpus of English writing, except that all spaces and punctuation are omitted, and we tried to infer a dictionary of English words from that.  We might naturally infer that "ARE", "THE", and "HER" are reasonable tokens, as they occur frequently as substrings.  All well and good so far.  However, we wouldn't want the algorithm to add "ARETHE" to the dictionary, even though "ARETHE" appears fairly frequently, as "ARETHE" is just the concatenation of the two tokens "ARE" and "THE".  I suspect it might be possible to detect this, because the frequency of "ARETHE" is not much more than the product of the frequencies of "ARE" and "THE", but I'm not sure.  Is there a clean way to handle this wrinkle?


texts: Are there any libraries for drawing a neural network in Python?

texts:
I'd like to render a neural network in Python to make sure it is set up properly.  I'd like to verify that it has the correct # of nodes, correct edge configuration, and perhaps even correct edge weights after training.  I'd like to visually debug simple examples by printing them to a PNG/JPG but do not know any libraries that do this.  Is there a library for drawing a neural network for visual debugging?

(Such a library would of course make assumptions on the data types/structures - e.g., assuming the neural network comes from Theano, TensorFlow, or some other library.  I'm flexible on the assumptions made.)


texts: Suggestions for labeling data for named entity recognition

texts: Is it good to label the data based on sub category than parent category? For example: for drugs data ... label the drugs dose as drug_dose or label the drug dose as different type of dose like injection, capsule, spray..which is a better option?


texts: Best way to scale across different datasets

texts:
I have come across a peculiar situation when preprocessing data.

Let's say I have a dataset A. I split the dataset into A_train and A_test. I fit the A_train using any of the given scalers (sci-kit learn) and transform A_test with that scaler. Now training the neural network with A_train and validating on A_test works well. No overfitting and performance is good.

Let's say I have dataset B with the same features as in A, but with different ranges of values for the features. A simple example of A and B could be Boston and Paris housing datasets respectively. To test the performance of the above trained model on B, we transform B according to scaling attributes of A_train and then validate. This usually degrades performance, as this model is never shown the data from B.

The peculiar thing is if I fit and transform on B directly instead of using scaling attributes of A_train, the performance is a lot better. Usually, this reduces performance if I test this on A_test. In this scenario, it seems to work, although it's not right.

Since I work mostly on climate datasets, training on every dataset is not feasible. Therefore I would like to know the best way to scale such different datasets with the same features to get better performance.

Any ideas, please.

PS: I know training my model with more data can improve performance, but I am more interested in the right way of scaling. I tried removing outliers from datasets and applied QuantileTransformer, it improved performance but could be better.


texts: Pytorch: How to implement nested transformers: a character-level transformer for words and a word-level transformer for sentences?

texts:
I have a model in mind, but I'm having a hard time figuring out how to actually code it in Pytorch, especially when it comes to training the model (e.g. how to define mini-batches, etc.). First of all let me quickly introduce the context:

I'm working on VQA (visual question answering), in which the task is to answer questions about images, for example:

So, letting aside many details, I just want to focus here on the NLP aspect/branch of the model. In order to process the natural language question, I want to  use character-level embeddings (instead of traditional word-level embeddings) because they are more robust in the sense that they can easily accommodate for morphological variations in words (e.g. prefixes, suffixes, plurals, verb conjugations, hyphens, etc.). But at the same time I don't want to lose the inductive bias of reasoning at the word level. Therefore, I came up with the following design:

As you can see in the picture above, I want to use transformers (or even better, universal transformers), but with a little twist. I want to use 2 transformers: the first one will process each word characters in isolation (character-level transformer) to produce an initial word-level embedding for each word in the question. Once we have all these initial word-level embeddings, a second word-level transformer will refine these embeddings to enrich their representation with context, thus obtaining context-aware word-level embeddings.

The full model for the whole VQA task obviously is more complex, but I just want to focus here on this NLP part. So my question is basically about which Pytorch functions should I pay attention to when implementing this. For example, since I'll be using character-level embeddings I have to define a character-level embedding matrix, but then I have to perform lookups on this matrix to generate the inputs for the character-level transformer, repeat this for each word in the question and then feed all these vectors into the word-level transformer. Moreover, words in a single question can have different lengths, and questions within a single mini-batch can have different lengths too. So in my code I have to somehow account for different lengths at both word and question levels simultaneously within a single mini-batch (during training), and I've got no idea how to do that in Pytorch or whether it's even possible at all.

Any tips on how to go about implementing this in Pytorch that could lead me in the right direction will be deeply appreciated.


texts: How to measure the contribution of each input channel to a network's accuracy?

texts: As the title says, let's assume we have some neural network. More specifically, a regression network in case that changes anything. It takes multiple 2D input channels. How could one measure how much each input channel is contributing to the accuracy of the network on a validation set, to know which channels to keep and which are not really necessary?


texts: (SOLVED) Power BI, Page level filter not working with many to one relation

texts:
EDIT: Solution found.

Turns out the relations were set to Cross filter direction: Single on all relations (and the one from Month table.Month to A.Month was not set to active).

Setting Cross filter direction: Both fixed the problem.

I have four tables A, B, Month table and Quarter table

A has a column A.Month, and many other columns with irrelevant data

B has a column B.Quarter, and many other columns with irrelevant data

Month table has columns Month table.Month and Month table.Quarter

Quarter table has a column Quarter table.Quarter

One to many relations are shown as this: "<"

Quarter table.Quarter < Month table.Quarter as there are multiple (3) months in one quarter

Month table.Month < A.Month as there are several rows of data for each month

Quarter table.Quarter < B.Quarter as there are several rows of data for each quarter

Here is a visualisation of the relations

On my dashboard, I have a visual for A, and a visual for B

What I'm trying to do is to apply a page level filter with Month table.Month, thinking that therefore, A should only show data for whatever month/months are selected in the filter, and B should show the data for the corresponding quarter/quarters.

Filter to January

Expectation:

A shows data for January
B shows data for Q1

Reality:

A shows data for all months
B shows data for all quarters

Switch page level filter from Month table.Month to Quarter table.Quarter and set filter to Q1

Expectation:

A shows data for January, February and Mars
B shows data for Q1

Reality:

A shows data for all months
B shows data for Q1

What am I missing?


texts: How does layer normalization work exactly?

texts: As far as I understand, layer normalization normalizes across all the features for fully connected layers. Does that mean that for each batch dimension we have to learn the normalization parameters? Also, what do we do at test time if our batch dimension is different?


texts: In Conditional Random Fields, is mandatory to use features related to following and preceeding tokens?

texts:
I am training a CRF classifier to classify document rows as a heading (1st level),  heading (2nd level) or simple text.
I am using Conditional Random Fields for their ability to account sequential aspects.
Reading some tutorials, I noticed that usually, among the features, there are some features related to the preceeding or following token.
[...]
    if i > 0:
        word1 = sent[i-1][0]
        postag1 = sent[i-1][1]
        features.update({
            '-1:word.lower()': word1.lower(),
            '-1:word.istitle()': word1.istitle(),
            '-1:word.isupper()': word1.isupper(),
            '-1:postag': postag1,
            '-1:postag[:2]': postag1[:2],
        })
    else:
        features['BOS'] = True
[...]

I wonder if the sequential aspect is learned from these features or is connate in CRF. In other words, do we need these features related to other tokens?


texts: Why do train, test, validation datasets need to have the same distribution?

texts:
I've found a lot of martial on how to deal with differently distributed train/test/validation data sets, however I'm struggling to find out why they they need to have the same distribution.

Can anyone explain to me in detail why these data sets need to have the same/similar distributions? or point me toward resources that explain it?


texts:
While I was studying, few years ago , one of the most interesting topic was evolution, genetic algorithms and neural networks. Many of the problems I faced could be solved by using that knowledge.

I assume that since that time world has found more interesting algorithms, do you recommend some books worthy of reading in that domain ?

Mainly I am looking a way to find patterns in huge amount of data.

Ex. Having energy consumption for few years for one building.

Lets have an algorithm that is trying to find all possible repeatings in many variations.

Obviously at the beginning it should find that at the weekend energy consumption is less then average between Mon and Fri, but is that possible that an algorithm would tell me sth like this ? :

Every Friday at third week in even months user sleeps for 3 hours and in uneven months 5 hours ?

Or the algorithm finds it self that user like to save energy so if previous month he sees that he spent more then usual , next month he is trying to spent less,

or lets assume that user eats breakfast at work but if he eats it at home then he will stay whole day at home, then lets assume that he has off day, then check energy usage after breakfast if is high, which means user is preparing to leave or is small which means that he wants to sleep and basically stays home.

So I was wondering if this possible to auto detect this kind of patterns ?

I am enthusiast of c# and interested in R.


texts: How does the naive Bayes classifier handle missing data in testing?

texts: Assume that a classier has been trained already (no missing training data), but a prediction has been requested based on an observation that does not include every feature. How can we handle this missing feature?


texts: learn information from text and resolve problem using transformers

texts:
Let's imagine that we have some question, like this: "x multiplied by x equals 9. What is x?"
For this easy question answer is +-3. I want to make AI model answer on questions like that.
To train model we have only corpus
Ex:
"If some variable in power one multiplied by itself and equals to some digit, then we have to get root square from this digit in order to find x". Model has to learn some data from text. Model can't just look for answers in text.
How to do model which could answer on this kind of questions?
I'm waiting for some tips.
UPD:
I heard that someone had made AI to resolve schrodinger equation. I want to reproduce their work, but for my own tasks ,which are more easy.


texts: Solving a system of equations with sparse data

texts:
I am attempting to solve a set of equations which has 40 independent variables (x1, ..., x40) and one dependent variable (y). The total number of equations (number of rows) is ~300, and I want to solve for the set of 40 coefficients that minimizes the total sum-of-square error between y and the predicted value.

My problem is that the matrix is very sparse and I do not know the best way to solve the system of equations with sparse data. An example of the dataset is shown below:

   y    x1  x2 x3 x4 x5 x6 ... x40
87169   14  0  1  0  0  2  ... 0
46449   0   0  4  0  1  4  ... 12
846449  0   0  0  0  0  3  ... 0
....

I am currently using a Genetic Algorithm to solve this and the results are coming out
with roughly a factor of two difference between observed and expected.

Can anyone suggest different methods or techniques which are capable of solving a set of equations with sparse data.


texts: Statistical Significance of precision WITHOUT negative labels

texts:
Let there be $n$ data points (examples) to be classified into 2 classes - either positive or negative. Let's say we have a model that outputs the probability of an example being of the positive class. I want to decide on an appropriate threshold for the probability - an example will be predicted as positive if the corresponding probability is more than the threshold.

The constraint is that I only know that $k$ of those examples are positives. I have no information on any true negatives. So for example if there are total 1500 examples, I might only know that 50 of them are positive class, and that's it.

If I set a threshold $T$, then I can count the no. of true positive examples whose probability score exceeds $T$. This gives me probability on the sample $P(>T\ |\ pos)$ where "$>T$" is the event of an example's probabilty score exceeding $T$. Then

$$P(pos\ |\ >T)=\frac{P(>T\ |\ pos)P(pos)}{P(>T)}$$

Now I know $P(>T)$ - I can just count the no. of examples exceeding the threshold. Assuming I have an estimate for $P(pos)$ as well, there's still the problem that $P(>T\ |\ pos)$ has been estimated on the sample and I don't know how it affects the final estimate of $P(pos\ |\ >T)$.

Doubt: Is there a way to calculate some measure of confidence for the final probability estimate that I calculate?


texts: Which Spark MLlib regression algorithm is suitable for numeric predictions based on non-numeric features?

texts:
I am working on Spark MLlib and have a project where I have to make predictions for numeric data based on non-numeric features. I am a bit confused about which regression algorithm to use from Spark MLlib library primarily due to being new at this. The algorithms present in Spark MLlib library are:

-linear models (SVMs, logistic regression, linear regression)
-naive Bayes
-decision trees
-ensembles of trees (Random Forests and Gradient-Boosted Trees)
-isotonic regression

Can anyone provide me some guidance as to which algorithm will be suitable for predictions for numeric data based on non-numeric features?


texts: Any research on segmentation of non-text contents out of (mostly) text-documents?

texts: Documents, especially technical ones often contain non-text content in blocks (code snippets, os commands etc). Is there any efficient way to identify the starting and ending lines of those blocks?


texts: How to distinguish between different values of a hyperparameter in communication?

texts:
From Wikipedia:

In machine learning, a hyperparameter is a parameter whose value is
used to control the learning process.

If we go by the definition of parameter in What's the difference between an argument and a parameter?, then a hyperparameter wouldn't be considered a value itself. Rather, it would be the spec for this value (for example, this value is a learning rate, or this value is a dropout). This also seems to agree with the way people use "hyperparameter" in communication.
There are a lot of times where I want to distinguish between particular values of a hyperparameter. Going by the question I linked, I could say something like hyperargument. For example, the hyperargument 0.001 is different than the hyperargument 0.01, even though they are under the same hyperparameter (let's say learning rate). This is what I do in my own codebase for variable names. For example, if I had this dict:
hyperarguments = {
    "learning_rate": 0.001,
    "dropout": 0.25,
}

I would iterate over it like this:
for hyperparameter, hyperargument in hyperarguments.items():
    ...

Is there a more common terminology to emphasize this distinction of values?


texts: Using Generative Adversarial Networks for a generation of image layer

texts:
Has anybody seen any application that would use GAN that would take input image and would output image of the same size, that could be used as a layer for the first image. The layer would contain in eg. point of interest in the input image. Would that even be a good practice of the GAN usage?

I am looking for articles, examples of application that used something similar.


texts: Understanding interplay between eval_metric, metric and first_metric_only parameters in LGBMClassifier

texts:
In python API of LGBMClassifier, the constructor takes parameters metric and first_metric_only. Their descriptions are as follows:

first_metric_only , default = false, type = bool
set this to true, if you want to use only the first metric for early stopping

metric , default = "", type = multi-enum, aliases: metrics, metric_types
metric(s) to be evaluated on the evaluation set(s)

The fit method also takes an argument eval_metric whose description is:

eval_metric (string, callable, list or None, optional (default=None)) – If string, it should be a built-in evaluation metric to use. If callable, it should be a custom evaluation metric, see note below for more details. If list, it can be a list of built-in metrics, a list of custom evaluation metrics, or a mix of both. In either case, the metric from the model parameters will be evaluated and used as well. Default: ‘l2’ for LGBMRegressor, ‘logloss’ for LGBMClassifier, ‘ndcg’ for LGBMRanker.

It is not clear what they mean by first metric only. I have sent in ['auc','binary'] as value to argument metric  and ['auc'] to eval_metric with first_metric_only set to True and yet it is using 'binary' for early stopping as well. I have always tried other combinations but I am not able to make clear what would happen in different scenarios
Can somebody please tell me what the exact interplay between these parameters is and what is meant by 'first metric'?


texts: Keras objective function shared between outputs

texts:
Is there any way to implement a loss function that is shared between outputs? I have a 2D image output and scalar classification that are both used by a single loss function.

I have attempted writing a function that returns a function, as in this comment, but I would need the input to the function to be the current training example. I also thought of using a merge layer, but that wouldn't work due to an incompatibility of the layer dimensions.

Does anyone know of a way to write such a loss function in keras?


texts:
I want to make an LSTM network and I have quite a long signal that I want to use as my training data

My X_train is a CSV-file which contains 12 signals with a length of 54 837 488
My y_train is an array that contains a One Hot encoded signal (9 categories) with a length of 54 837 488

If I try to upload the CSV-file as a data frame or as an array in Python I exceed the limit of RAM, so my thought is to use fit_generator(). I have used this before when making image models but then I just used some prebuild generators, but I can't really find any prebuild generators for signals so I decided to try to make one myself.

def generate_data_to_model(y_train):
    while True:
        with open("/mypath/myData.csv") as f:
            for line in f:
                x= line.rstrip('\n').split(",")
                x= np.asarray(x)
                x=x[1:]
                x= x.reshape(1,1,12)
                yield (x, y_train[line])

model = Sequential()
model.add(LSTM(32, input_shape=(1, 12)))
model.add(Dense(32, activation='relu'))
model.add(Dropout(0.5))
model.add(Dense(9, activation='softmax'))

model.compile(loss='categorical_crossentropy', optimizer="Adam", metrics=['acc'])

model.fit_generator(generate_data_to_model(y_train),
                    steps_per_epoch=1, epochs=2, verbose=1)

When I start training I get this error:

<ipython-input-29-33b2195fea44> in generate_data_to_model(y_train)
      8                 x= x.reshape(1,1,12)
 ---> 9                 yield (x, y_train[line])

  only integers, slices (:), ellipsis (...), numpy.newaxis (None) and integer or boolean arrays are valid indices


texts: How to represent a user who hasn't churned in training data

texts:
I am building a file with sample data that has a bunch of variables:

date, customer_id, amount_spent, number_of_transactions, time_since_last_transaction etc. that i am mapping against days_to_churn

I will train my model using Keras to map the emboldened variables to the italicised days_to_churn. However there are many instances where a user is an active subscriber and this value is blank.

How would I go about incorporating this to ensure that I am not excluding active people from my churn calculations?


texts: sklearn ColumnTransformer creates new columns in output when there are overlapping columns between steps

texts:
I need to process some dataframe columns in different steps using ColumnTransformer.
The first step process the date columns (timestamp) imputing missing values and the second step applies scaling to all the numeric columns (including the dates columns). In output I get a number of columns which is the sum of the numeric columns and the dates columns, but the dates columns are a subset of the numeric columns so this is not correct.

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder

dates_columns = ['ts_1', 'ts_2']
numeric_columns = ['ts_1', 'ts_2', 'n_1', 'n_2']

column_transformer = ColumnTransformer([
    ('imputer_dates', SimpleImputer(strategy='median', missing_values=0), date_columns),
    ('scaler', StandardScaler(), numeric_columns)
])

X_transformed = column_transformer.fit_transform(X)
print(X_transformed.shape) # Got 6 columns, but it should be 4

How to fix this?


texts: Guidelines to debug REINFORCE-type algorithms?

texts:
I implemented a self-critical policy gradient (as described here), for text summarization.

However, after training, the results are not as high as expected (actually lower than without RL...).

I'm looking for general guidelines on how to debug RL-based algorithms.

I tried :

Overfitting on small datasets (~6 samples) : I could increase the average reward , but it does not converge. Sometimes the average reward would go down again.
Changing the learning rate : I changed the learning rate and see its effect on small dataset. From my experiment I choose quite big learning rate (0.02 vs 1e-4 in the paper)
Looking at how average reward evolve as training (on full dataset) goes : Average reward significantly does not move...


texts: Detection of Anomalous Sequential User Behavior

texts:
I have a dataset containing a set of normal user sessions. Each session contains a suite of ordered user requests on N system resources {R1, ..., RN}.

I want to design a continuous authentication algorithm, by confirming the user identity at each request command. More precisely, I don’t to let the user complete the whole session (all commands sequence) to authenticate him, but I want to do this at each resource requested command based on his previous normal sequences in the dataset.

How can I build a model over normal sequences to detect anomalous ones in Resource-by-Resource fashion ?

I found a lot of key concepts (Sequential Pattern Mining, Sequence Anomaly Detection, …), but I am completely confused about these concepts

I would like to know how to begin and which ML algorithm(s) I should use.


texts: Effective way to find similarity between utterance(short text) and question(long text)

texts:
The challenge I have is a bunch of questions(long text) that are closely matching with an user utterance (short text). I have tried cosine similarity & Tf-iDF, BM25,Jaccard similarity, etc., but none of these give desired results.
For instance, my question list is
a) how do I transfer from XYZ account
b) will I be able to transfer from XYZ account with zero fees
c) how do I receive some money from XYZ account
utterance - "transfer XYZ account" should result in close similarities between all these.
However an utterance - "transfer XYZ account with some fees" should result in b alone (with other two questions having very less similarity scores)
Another challenge that I face is ordering of these entities. From A --> B is treated similarly as B --> A.


texts:
So I am currently trying to implement an LSTM on Pytorch, but for some reason the loss is not decreasing. Here is my network:
class MyNN(nn.Module):
    def __init__(self, input_size=3, seq_len=107, pred_len=68, hidden_size=50, num_layers=1, dropout=0.2):
        super().__init__()

        self.pred_len = pred_len

        self.rnn = nn.LSTM(
            input_size=input_size,
            hidden_size=hidden_size,
            num_layers=num_layers,
            dropout=dropout,
            bidirectional=True,
            batch_first=True
        )

        self.linear = nn.Linear(hidden_size*2, 5)

    def forward(self, X):
        lstm_output, (hidden_state, cell_state) = self.rnn(X)

        labels = self.linear(lstm_output[:, :self.pred_len, :])

        return lstm_output, labels

And my training loop
LEARNING_RATE = 1e-2

net = MyNN(num_layers=1, dropout=0)

compute_loss = nn.MSELoss()

optimizer = optim.Adam(net.parameters(), lr=LEARNING_RATE)

all_loss = []
for data in tqdm(list(train_loader)):
    X, y = data

    optimizer.zero_grad()

    lstm_output, output = net(X.float())

    # Computing the loss
    loss = compute_loss(y, output)
    all_loss.append(loss)
    loss.backward()

    optimizer.step()

# Plot
plt.plot(all_loss, marker=".")
plt.xlabel("Epoch")
plt.xlabel("Loss")
plt.show()

And this is what I got

I have been trying to look for what the hell I am doing wrong but I have no idea. Also, before I used a keras LSTM and it worked well on the dataset.
Any help?
Thanks!


texts: How to further Interpret Variable Importance?

texts:
Forgive me if this is a duplicate question, I haven't found anything that answers my question specifically after searching for a while.

I have a dataset which I'm using to predict mobile app user retention, using the RandomForestClassifier in the SciKit Learn package. I'm pleased with the accuracy I'm getting and I'm planning on including a number of other metrics including precision, recall, Matthew's Corr Coefficient etc. I'm pretty sure the model is good.

The key thing that I'm interested in here are the features themselves. I want to know what is contributing to my user churn. I have extracted the feature importances and plotted a nice looking graph, but now I'm stuck. I'd ideally like to know how each variable influences the churned/not churned outcome. The problem with GINI feature importances is that I can see which ones are most influential, but for example with continuous variables I want to know at which value the RF found best to split on. I don't need to see this for every feature as I have 70+ only the most 'important' ones.

I saw a very nice decision tree plot here, but cannot find any way of reproducing something similar using scikit learn. I'm open to other suggestions. Thanks in advance :)


texts: How do I explain the relationship between Health and Weight?

texts:
I have two columns in my dataset:  Health and Weight,  both being of numeric type:

    Health<-number of days when health is not good,
    Weight<-weight

All I want to check if there  is a relationship between Health and Weight. In other words, does a Weight increase yield an increase of the number of days  where Health is not good, or the  converse?  I just want to check the relation between those two columns in dataset by plotting some graphs.

Here my sample dataset:

| Health     | Weight      |
|:-----------|------------:|
| 0          |      30     |
| 3          |      63     |
| 2          |      31     |
| 10         |      169    |
| 1          |      9      |
|0           |     139     |


texts: Do all layers have the same computational complexity in a ResNet?

texts:
Reading the ResNet paper, paragraph 3.3:

  The convolutional layers mostly have 3×3 filters and follow two simple
  design rules: (i) for the same output feature map size, the layers
  have the same number of filters; and (ii) if the feature map size is
  halved, the number of filters is doubled so as to preserve the time
  complexity per layer.

The network uses convolutional layers with stride 2 to halve the feature map size, but this divides both the width and the height of the feature maps, so the total area is divided by 4, not by 2.

So I am guessing the time complexity per layer is halved every time the feature maps are halved. Did I miss something?


texts: Dealing with time series data which has multiple observations for the same timestamp

texts:
I have a time series data in Python 3 as follows:

Date                `Weekly_Sales`
2010-05-02              3400
2010-05-02              5600
2010-05-02`             4590
2010-05-02              5800
2010-05-12              2380
2010-05-12              6700
2010-05-12              3700

The time series  is not continuous as there are multiple observations of the same date.I'm trying to forecast sales in python using ARIMA but my ACF and PACF plot shows that there is no corelation between the lags.Also if i run the dickly fuller test to test stationarity,my system freezes.

How can I fix this?


texts: ValueError: ('The truth value of a Series is ambiguous after applying if/else condition in Pandas data frames

texts:
I want to create a new variable for the dataframe details called lower after iterating through multiple data frames.

list1 is a list of string values of a column named variable_name in the details.
vars_df is another data frame with 2 columns, namely variable_name and direction. Both columns contain with string values. vars_df.shape = (19,2)
Some values of variable_name in vars_df are present in list1 as well as data_set.
data_set.shape = (32,107).df.shape = (96,1)

The following code, which aims to do the above:
    def get_value(df,list1):
        if ((vars_df['variable_name']) in list1) & (vars_df['direction'] =='up'):
            data = data_set[df['variable_name']]
        else:
            data = data_set[df['variable_name']]-20
            if data < 0:
               data = 0
            else:
               data = data
       return data

details['lower'] = details.apply(get_value,list1=li,result_type='expand',axis=1)

produces this error:
ValueError: ('The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().', 'occurred at index 0')
I referred the question.Truth value of a series is ambiguous Still I can not rectify the error


texts: Convolutional neural network block notation

texts:
The paper by He et al. "Deep Residual Learning for Image Recognition" illustrates their residual network in Figure 3 as follows:

I am not a neural network expert, so could somebody please explain to me what the highlighted notation above "3x3 conv, 256, /2" means? The first part is clear (convolutional neural network with a 3x3 pixel window), but what is the "256" and the "/2"?


texts: Evaluating a Multi-Label Classification model

texts:
I currently have a multi-label classification problem, for which I am using keras to build a neural network as follows:
n_cols = dataset.shape[1]
print(n_cols)

model = Sequential()
model.add(Dense(128, activation='relu', input_shape=(n_cols,)))
model.add(Dense(64, activation='relu'))
model.add(Dense(26, activation='sigmoid')) # Sigmoid for multi-label classification

sgd = SGD(lr=0.1, decay=1e-6, momentum=0.5, nesterov=True)
#RMSprop
model.compile(loss='binary_crossentropy', optimizer='RMSprop', metrics=['accuracy'])

model.summary()

## Fit the model ##
early_stopping_monitor = EarlyStopping(patience=20)
history = model.fit(dataset, labels, validation_split=0.33, epochs=30, callbacks=[early_stopping_monitor])

plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['Train', 'Test'], loc='upper left')
plt.show()

I was informed that for multi-label classification, we use binary_crossentropy for the loss while having sigmoid for activation in the final layer (output layer). However, with this I am getting a resulting accuracy and val_accuracy of ~0.0931 and ~0.0937 respectively.
For the multi-label classification, is using the accuracy metric the best fit? I've looked around and some suggest that other metrics such as binary_accuracy may be better..
So the question is, how can one best evaluate the multi-label classification?
EDIT: For reference, I have 26 label columns in my target "classes" and the dataset consist of 21 columns. The entire dataset the model is trained on has ~82k samples.


texts: Directed graph with repulsor and attractor edges

texts:
I'm attempting to model the relationships (likes and hatreds) between various mercenaries to reduce friction after team formation.
I'm using sigma.js as graph drawing library and forceAtlas2 as force-directed layout algorithm and having some success (see images below) in modeling just the Likes.
Mercenaries form the Nodes and their Likes for the directed Edges.
My understanding is the Nodes repel each other and the Edges cause attraction. I can color the Edges differently for Likes and Hatreds, but what I'd really like is to change the strength of attraction (including to negative for hatreds). But I don't know what that would be called, limiting my solution seach.
Is what I'm asking possible / mathematically sound? Am I using appropriate tools for the problem I'm describing?
Advice would be much appreciated, as my background isn't in math / data science.


texts: Solution for TF-IDF Vectorization in Angular project?

texts:
While making an Angular project to use my text-classification model on unseen data, i struggle in finding a way how to transform text to TFIDF features.

Anyone faced same issue? Maybe a solution on how to complete this in another way?


texts: Scikit-learn pipeline with scaling, dimensionality reduction, average prediction of multiple regression models, and grid search cross validation

texts:
I would like to use a sklearn pipeline doing this :

( - ) scale the data ( StandardScaler )

( - ) reduce dimensionality ( PCA )

( - ) make a prediction with GradientBoostingRegressor() and GridSearchCV() ( to get the model with best parameters from grid )

( - ) make a prediction with RandomForestRegressor() and GridSearchCV() ( to get the model with best parameters from grid )

( - ) take the mean of both predictions

but I cannot figure out how to proceed.

Do I have to ( scale & predict ) 2 times or can I scale and then predict 2 times ?

Thks.


texts: Reinforcement Learning with static state

texts:
Can Q Learning work with a static state for each step?

What I mean by that is that the actions do not influence the following state at all. The episodes just iterate over the same data over and over again. Of course the different actions lead to different rewards, but is Q Learning the right concept to use in this case? What other types could you suggest?


texts: How to measure the performance of an imputation technique

texts:
I would like to know how I can measure the performance of an imputation technique. I have read a lot about this. Most literature on the web are applying a classifier after the data has been completed. So this classifier will be used in order to make predictions. However, I am not interested to use such classifier to make predictions.

I would like to know how good an imputation method performs on a dataset. We can measure this for example my variance, mean, mean squared error and so on. I would like to know if there are similar techniques to measure the quality of the data. I will use the dataset for descriptive analysis(not for predictive analysis, e.g. training classifier)

Please correct me if I am thinking in the wrong context/corner. Thanks in advance.

Best Regards


texts: RandomForestRegressor intermittantly returning a single prediction

texts:
BACKGROUND

I have a RandomForestRegressor from scikit-learn which, for each example row, takes in four float features and predicts a float target. All numbers are scaled and lie within "reasonable" ranges. To be specific, the target float is simply the next value of one of the features. I.e., it's a time-series prediction on one of the features (which I'll refer to as the "driving" feature below).

PROBLEM

Most of the time, the model trains and serves correctly, attaining satisfactory mean-squared errors. However, every once in a while, and with the exact same data, the model learns that the target value is simply the previous value of the driving feature. I.e., all the model learns to do is "memorize" the previous value of the driving feature---so no actual fitting per se is performed.

QUESTION

Why does this happen? The code, data, and execution environment are held constant. It's really the model that behaves this way intermittently. I surmised that initialization (cf. random_state) may have something to do with this, but I feel that I don't want to introduce any kind of bias by arbitrarily setting a seed. The problem isn't so much reproducibility, but rather the breakdown of the learning into a simple memorization model.


texts: What ML architecture fits fixed length signal regression?

texts:
My problem is of  regression type -
How to estimate a fish weight using fixed length signal (80 data points) of the change in resistance when the fish swim through a gate with electrodes (basically 4 seconds of the fish passing at 20Hz logger)?
It is a spike shaped-signal, height and width depends on the size of the fish, its speed and proximity to the gates edges and probably other things like the water salinity and temperature.
I have a data set of 15 different weights, each with 20-110 samples, each with 2 spikes for the 2 sets of electrodes I use for measurement (using 2 sets can help determine where the fish is heading).
Here is an example of resistance readout of 340 gram fish experiment:

And here is an example of the extracted spikes from the same 340 gram fish experiment:

As you can see, there is a significant variance, which led me to look for a Neural Network approach that can get such signal as an input and estimate the fish weight.
Do you know of such "State of the Art" network that does that?
What would you try?
Maybe different ML technique?
Thanks!
Edit:
The data presented is post-processing, I extract the spikes using this python code (attached) so some of the noise is cleaned.
I'm not sure as to how to clean it any better since the experimenter didn't record when a fish goes through the gate - all we have is the electrodes signal to deduce that a fish passed through.
# extracting the spikes
def get_spikes(data_series_elc1, data_series_elc2, signal_meta):
    window_size = int(signal_meta['freq'])*4
    half_window = int(window_size/2)

    std = np.std(data_series_elc1)
    p10 = np.quantile(data_series_elc1, 0.9)
    spikes = []
    i = 0
    while i < len(data_series_elc1)-half_window:
        if data_series_elc1[i] > p10:
            #find next max to fix as the center
            max_indx = np.argmax(data_series_elc1[i:i+window_size]) half_window:i+max_indx+half_window])
            spike_list = [[data_series_elc1[i+max_indx-half_window:i+max_indx+half_window]],[data_series_elc2[i+max_indx-half_window:i+max_indx+half_window]]]
            if len(spike_list[0][0])==window_size:
                spikes.append(spike_list)

            i = i+max_indx+half_window
        else:
            i = i+1
    print('Number of Spikes: ',len(spikes))
    return spikes

Also, I extract features like max, width, integral and Gaussian fit but a linear regression model only gets me ~R^2=0.6 => a mean error of ~103 gram over all fish
[100., 144., 200., 275., 339., 340., 370., 390., 400., 404., 480., 500., 526., 700., 740., 800., 840.], which is quite a large error.
A vanilla fully connected neural network gets about the same.
model = keras.Sequential()
model.add(keras.Input(shape=(80,)))
model.add(layers.Dense(40, activation="relu"))
model.add(layers.Dense(10, activation="relu"))
model.add(layers.Dense(1))

So I'm looking to improve these results, any ideas?


texts: Data preprocessing : Aggregation, feature creation, or else?

texts:
I have a problem to name data processing step.

I have an attribute that contain string or null. I want to change the record of an attribute to 0 if null and 1 if not null. What preprocessing step name?
I have an attribute that contain 200 tweets of a twitter user. I make a new attribute that describe URL ratio in 200 tweets (number of URL in 200 tweet / 200). is it aggregation or feature creation or else?


texts: How to sort of "average" data without smoothing?

texts:
I'm trying to figure out a method to perform some kind of average on my data, without smoothing the data. In my case, my data is a waveform, which means amplitude as a function of time.
The problem with method such as the moving average, is that the result is a smoothed version of the original data, which means losing important information like sharp edges, which I need to perform my analysis.

Do you have any advice?


texts: Random forest confusion matrix encountered invalid values

texts:
I am doing classificaion using random forest classifier in python (scikit learn).
I have many different databases, each one has 33 observations and the prediction is based on 600 columns.
The script is iteration which run the classifier and then create confusion matrix for each.
When I run the script it works but for some databases I get the next error message:

:25: RuntimeWarning: invalid value encountered in true_divide   matrix = matrix.astype('float') /
matrix.sum(axis=1)[:, np.newaxis]

also the results look weird:

I understand that this might happen if I have Null values in my data, but I have used dropna to make sure that there are no null values:
df=df.dropna(axis=0,how='any')

This is my script for the confusion matrix iteration:
for h in dfch:
    print('Hour:',h)
    print('')
    list_dates=dfch[h]['date'].unique()
#     print(h)
#     print(list_dates)
    for d in list_dates:
        print('date:',d)
        print('hour:',h)
        dfhd=dfch[h]
        dfhd=dfhd.loc[dfhd['date']==d]
        print('database size for hour',h,'date',d,'is',len(dfhd))
        X=dfhd.iloc[:, 4:]
        y=dfhd.iloc[:,2:3]
        #split the data
        X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
        #reshape the y_train to fit the the model
        y_train=y_train.values.ravel()
        #fit the model
        rfc.fit(X_train,y_train)
        rfc_pred=rfc.predict(X_test)
        print('')
        # Get and reshape confusion matrix data
        matrix = confusion_matrix(y_test, rfc_pred)
        matrix = matrix.astype('float') / matrix.sum(axis=1)[:, np.newaxis]
        # Build the plot
        plt.figure(figsize=(16,7))
        sns.set(font_scale=1.4)
        sns.heatmap(matrix, annot=True, annot_kws={'size':10},cmap=plt.cm.Greens, linewidths=0.2)
        # Add labels to the plot
        class_names = ['high','medium','low']
        tick_marks = np.arange(len(class_names))
        tick_marks2 = tick_marks + 0.5
        plt.xticks(tick_marks, class_names, rotation=25)
        plt.yticks(tick_marks2, class_names, rotation=0)
        plt.xlabel('Predicted label')
        plt.ylabel('True label')
        plt.title('Confusion Matrix for Random Forest Model')
        plt.show()

        score=rfc.score(X_test,y_test)

My question : How can it happen that it will be divided in 0/null? (there is no null in the databases), and how does it still display the confusion matrix if it fails due to dividing by 0? How canI solve it?


texts: On-the-fly seq2seq: starting translation before the input sequence ends

texts:
By now anybody even remotely familiar with RNNs has been exposed to the famous figure representing variants of such networks:

However one type that is missing is one where there would be a partial overlap in time between the input and the output.

For example, in machine translation, a commonly presented solution is to encode the input sequence entirely, and then, decode it. (That corresponds to the fourth variant above).

However, human translators typically do not wait till the end of a sentence to start translating: they would start as soon as they estimate that they have enough information for it.

Is there any literature on the subject, and what would be the key idea to train such a network? One difficulty I see is that in an RNN world, time is discrete, and therefore optimizing over the starting time would be difficult to achieve using SGD.


texts: How to present longitudinal data to LSTM for multiclass prediction

texts:
I need to implement a deep learning algorithm to predict an ordinal value, called 'Entity', using longitudinal health records data. I read a few articles and guides but I couldn't find a clear explanation or example on how to organize input data; the only thing that I've understood is that I need to use an LSTM node which is designed exactly for this kind of problem.
To clarify let me bring an example, let's say that I have this table:

ID
N1
C1
B1
Out

2
25
3
0
1

2
32
4
1
2

2
52
1
0
1

3
12
2
1
2

3
56
4
0
1

3
73
3
1
3

3
12
2
1
2

The table contains variables of different kinds like numerical, cardinal, and boolean; I need to teach my algorithm to predict the future output with longitudinal data (for example if I input the first two rows with ID=2 the output should be 1 which is in the third row with ID=2).
The rows with the same ID are different data from the same person, checked at different times.
I've understood that LSTM are layers useful when there is the need to catch time relations and the input is in shape (samples, timesteps, features); given that i have few doubts about how to treat and model the training data:

My dataset is not homogeneous in the timesteps dimension (persons may have different number of rows), but I think it's ok to have tensors with different timesteps length, is that correct?
if the answer at the previous question is yes, can i train my network with tensors of growing dimension of the same patient? For istance(notice that i want to predict the future Out):

train1 x:[25, 3, 0] -> label:2;
train2 x:[[25, 3, 0],[32, 4, 1]] -> label:1;

BONUS) Since is a complicated problem I was even wondering if I am looking at it in the wrong way and I should treat it like a kinda-regression problem like the one in that example https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/
To give a different perspective of the problem let's say doctors classify Alice by looking at the current situation and at previous visits, I would like to implement this behaviour in an algorithm so that when I have Bob current situation and clinical history I can make a prediction for him too based on the "rules" that i learned from Alice examples.
The alternative behaviour, that I expressed in the BONUS question, is if I want to reproduce the learning pattern of a Doctor which look at a graph like the one in the figure below and learn how to predict the trend of the y value in another patient based on the "rules" learned in Alice's graph.

I'm implementing that on Python using Keras, but I am stuck in this preprocessing phase and I can't proceed.


texts: Why bias is not considering in Regularization?

texts:
Most of the Regularization (L1, L2 ) techniques focused mostly on the weight term only .But Regularization is   not considering Bias.From my understanding  large bias doesn’t make a neuron sensitive to its inputs in the same way as having large weights and vice versa .which means Bias is also important as weight .

Can any one help me why Bias is not considering?


texts:
I am trying to add multiple new rows for 3 new factor level in an existing data frame. Please refer to sample data for an example. My starting data frame has 18 levels for col1 and all 12 months for column mon and past 20 years for year. I then impute values and add new columns, however I need new factors to be added for further analysis.

For each mon and year combination, a new level should exist.

Sample df:

col1 <- c(rep("a",4),rep("b",4))
col2 <- c(1:4)
mon <- c(rep(c("Jan","Feb", "Mar","Apr"), 4))
year <- c(rep("2016",8), rep("2015",8))
df <- as.data.frame(cbind(col1,col2,mon,year))

head(df,8)  # edited to make it readable

  col1 col2   mon year
1 a    1      Jan 2016
2 a    2      Feb 2016
3 a    3      Mar 2016
4 a    4      Apr 2016
5 b    1      Jan 2016
6 b    2      Feb 2016
7 b    3      Mar 2016
8 b    4      Apr 2016

Expected Output

   col1 col2   mon year
1  a    1      Jan 2016
2  a    2      Feb 2016
3  a    3      Mar 2016
4  a    4      Apr 2016
5  b    1      Jan 2016
6  b    2      Feb 2016
7  b    3      Mar 2016
8  b    4      Apr 2016
9  c    NA     Jan 2016  # New level c for each mon and year
10 c    NA     Feb 2016  # New level c for each mon and year
11 c    NA     Mar 2016  # New level c for each mon and year
12 c    NA     Apr 2016  # New level c for each mon and year

How do I go about reaching the expected df?


texts: is neural networks an online algorithm by nature?

texts:
I have been doing machine learning for a while, but bits and pieces come together even after some time of practicing.

In neural networks, you adjust the weights by doing one pass (forward pass), and then computing the partial derivatives for the weights (backward pass) after each training example - and subtracting those partial derivatives from the initial weights.

in turn, the calculation of the new weights is mathematically complex (you need to compute the partial derivative of the weights, for which you compute the error at every layer of the neural net - but the input layer).

Is that not by definition an online algorithm, where cost and new weights are calculated after each training example?

Thanks!


texts: How do I assign specific values to categorical variables

texts: I have a Pandas data frame with columns within a survey with the following categorical values -  "Increased, Decreased, Neutral". My question is how can I assign specific numerical values to these categorical values, namely +1 for Increased, -1 for Decreased and 0 for Neutral.


texts: Sklearn tfidf vectorize returns different shape after fit_transform()

texts:
I'm new to ML and trying out basic samples using sklearn. I have achieved converting text (single dimension) to numbers using TF-IDF and got the predictions correct.

Now I have a different use-case to predict Work Hours based on Marital-Status and Relationship Status so my training data is now two dimension.

I have DataFrame total of 24420 rows * 2 columns of shape --> (24420, 2)

After this DataFrame is been passed to TfidfVectorizer.fit_transform() function it turns to shape --> (2, 3) Not sure why!

And while training mnb.fit(x_train_tf, y_train) it returns an error

  ValueError: Found input variables with inconsistent numbers of samples: [2, 24420]

You could see the initial DataFrame shape of 24420*2 it transposed to 2*24420

All code is up here.

Questions:

Why after fit_transform() the shape changes to (2,3) what happened to (24420,2) data? What does this denotes?
Why does the ValueError: Found input variables with inconsistent numbers of samples [2, 24420] occurs?
Why initial DataFrame shape changed to 2*24420?

What is wrong with the implementation?


texts:
Consider the following code in Keras for building a LSTM model.

    model = Sequential()
    model.add(LSTM(30, input_dim=22,return_sequences=True, init = 'glorot_uniform'))

    model.add(TimeDistributed(Dense(1)))
    model.compile(loss='mean_squared_error', optimizer='Nadam')

    model.fit(train3d, trainY3d, nb_epoch=100, batch_size=8)

As one can see there is only 1 LSTM layer. I see there are 14 matrices as output. If you check this link you would notice that there are 4 triples (Input gate, Forget gate, Cell State, Output gate) of W, U (parameter matrices) and b (bias vector) each for the LSTM layer. And then there are 2 matrices W and b for the dense layer.

My question is let's say for this case of 1 layer LSTM, is there a way to attribute 100% of Y to the impact for each input feature $X_i \forall i$.


texts: What circumstances causes two different classifiers to classify data exactly like one another

texts:
Okay, here is the background:
I am doing text mining, and my basic flow is like this:
extract feature (n-gram), reduce feature count, score (tf-idf) and classify. for my own sake i am doing comparison between SVM and neural network classifiers. here is the weird part (or am i wrong and this is reasonable?), if i use 2gram the classifiers' result (accuracy/precision) is different and the SVM is the better one; but when i use 3-gram the results are exactly the same. what causes this? is there any explanation? is it the case of very separable classes?


texts: Architecture for multivariate multi-time-series model where some features are TS specific and some features are global

texts:
I'm looking to build a time series model (using a TCN or a LSTM)  with $N$ different series, each of which has $P$ series-specific features $\mathbf{X}$.  My input array is of dimension $N \times t \times P$, where $t$ is the number of time steps.

I've also got features $G$, which are constant across all time series.  For concreteness, imagine I'm predicting city-level ice cream sales with weather data, and I also want to use GDP growth as a predictor.  GDP growth is national.  A simple approach could be to augment $\mathbf{X}$ with $G$, adding 1 to the dimension of $P$.  Then my forecast output for the next period would be $N \times 1 \times P+1$, which is no good because there is a GDP forecast for each city, when in reality GDP growth is common across cities (when measured nationally).  I suppose that I want two outputs -- one of shape $N \times 1 \times P$, and the other of shape $1 \times 1 \times 1$, a scalar (if $G$ is of dimension $t \times 1$).

Here's a dummy example in which time is a global variable, but it is constant across all series.  (Let's just assume for the moment that time isn't exogenous, but rather something to include in a multivariate forecast).

import numpy as np
import matplotlib.pyplot as plt
from keras.models import Model
from keras.layers import Input, Conv1D, Dense
from keras.optimizers import Adam

time = np.array(range(100))
brk = np.array((time>40) & (time < 60)).reshape(100,1)
B = np.array([5, -5]).reshape(1,2)
np.dot(brk, B)
y = np.c_[np.sin(time), np.sin(time)] + np.random.normal(scale = .2, size=(100,2))+ np.dot(brk, B)

plt.plot(time, y[:,0])
plt.plot(time, y[:,1])

# Temporal convolutional network
n_filters = 2
filter_width = 3
dilation_rates = [2**i for i in range(5)]
inp = Input(shape=(None, 2))
x = inp
for dilation_rate in dilation_rates:
    x = Conv1D(filters=n_filters,
               kernel_size=filter_width,
               padding='causal',
               activation = "relu",
               dilation_rate=dilation_rate)(x)
x = Dense(2)(x)

model = Model(inputs = inp, outputs = x)
model.compile(optimizer = Adam(), loss='mean_squared_error')
model.summary()

def shift5(arr, num, fill_value=np.nan):
    result = np.empty_like(arr)
    if num > 0:
        result[:num] = fill_value
        result[num:] = arr[:-num]
    elif num < 0:
        result[num:] = fill_value
        result[:num] = arr[-num:]
    else:
        result = arr
    return result

X = y.reshape(2,100,1)
X = np.concatenate([X, np.concatenate([time.reshape(100,1),time.reshape(100,1)], axis = 1).reshape(2,100, 1)],
                    axis = 2)
X_tr = X[:,:95,:]
X_te = X[:,5:,:]

history = model.fit(X_tr, X_te,
                batch_size=2,
                epochs=10,
                verbose = 1)

How would I modify this architecture to have two inputs and two outputs, with both input and output having local and global components?


texts: Finding bounding box coordinates in Object Detection

texts:
In some of the OpenCV implementations for object detection , I don't understand how the co-ordinates of the bounding box of an object  are extracted from the image.
For example , In Object Detection using Deep Learning
There is a snippet of code I don't understand .
# loop over the detections
for i in np.arange(0, detections.shape[2]):
    # extract the confidence (i.e., probability) associated with the
    # prediction
    confidence = detections[0, 0, i, 2]
    # filter out weak detections by ensuring the `confidence` is
    # greater than the minimum confidence
    if confidence > args["confidence"]:
        # extract the index of the class label from the `detections`,
        # then compute the (x, y)-coordinates of the bounding box for
        # the object
        idx = int(detections[0, 0, i, 1])
        box = detections[0, 0, i, 3:7] * np.array([w, h, w, h])
        (startX, startY, endX, endY) = box.astype("int")
        # display the prediction
        label = "{}: {:.2f}%".format(CLASSES[idx], confidence * 100)
        print("[INFO] {}".format(label))
        cv2.rectangle(image, (startX, startY), (endX, endY),
            COLORS[idx], 2)
        y = startY - 15 if startY - 15 > 15 else startY + 15
        cv2.putText(image, label, (startX, y),
            cv2.FONT_HERSHEY_SIMPLEX, 0.5, COLORS[idx], 2)

In the line , box = detections[0, 0, i, 3:7] * np.array([w, h, w, h]) , Can someone please tell me why we multiply with the width and height of the image to get the co-ordinates of the box ? Doesn't the output detections[] itself give the co-ordinates ?
Now , my questions are , If 'detections' doesn't directly output the co-ordinates , what else does it output ?
And , Why is it required to multiply with the width and height of the image ?
Can someone please clear my doubts in a simple and effective manner ?


texts:
I'm trying to plot three heatmaps in a vertical column using Seaborns subplot method.

import seaborn as sns
initalCorr = inputX.corr()
secondaryCorr = inputX_corr.corr()
finalCorr = inputX[selected_columns_pvalue].corr()

fig, axs = plt.subplots(3, figsize = (15,45))
fig.suptitle('Heatmaps of correlation between features selected for training',
fontsize = 20)

sns.heatmap(data = initalCorr, center = 0, square = True, ax = axs[0])
axs[0].set_title('Correlation between all features', fontsize = 20)

sns.heatmap(secondaryCorr, center = 0, square = True, annot = True, linewidths = .5, fmt = '.2f', ax = axs[1])
axs[1].set_title('Correlation between features slected for P-Value analysis', fontsize = 20)

sns.heatmap(finalCorr, center = 0, square = True, annot = True, linewidths = .5, fmt = '.2f', ax = axs[2])
axs[2].set_title('Correlation between finally selected features', fontsize = 20)

The graphs themselves plot fine but the title appears out of place relative to the plots.

How do I make the title directly above the heatmaps?

Additional:
seaborn version 0.9.0
matplotlib version 3.0.3


texts: Multiple filtering pandas columns based on values in another column

texts:
I have a pandas dataframe df1:

Now, I want to filter the rows in df1 based on unique combinations of (Campaign, Merchant) from another dataframe, df2, which look like this:

What I tried is using .isin, with a code similar to the one below:

df1.loc[df1['Campaign'].isin(df2['Campaign']) &
        df1['Merchant'].isin(df2['Merchant'])]

The problem here is that the conditions are independent eg : I want to check if (A,1) from df2 is in df1, but with the above condition, since I am checking all the list, not row by row, it would return all rows in df1 where Campaign column is A OR Merchant column is 1.

Do you have any suggestion for this multiple pandas filtering?


texts: IOU accounting for the difference of the damage degree in GT and prediction

texts: I have a model of a skin disease condition. It takes a skin image and predicts areas affected by edema. Skin can be in one of four degrees, so each pixel is assigned a value. Healthy: 0, mild: 1, moderate: 2, severe: 3. I want to compare ground truth mask and predicted mask. Would there be only two degrees (healthy vs disease), I would use IOU. However, I want to account for the difference of the GT and predicted damage degrees. If the IOU value is the same the metrics should be higher if GT and predicted degree matches and lower if GT predicts mild degree, while the model predicts severe damage. What is the right way to account for IOU and the difference in damage degrees?


texts:
I am about to dive into a long NN research project and wanted a push in the direction of Pylearn2 or TensorFlow?  As of Dec 2015 has the community started to lean one direction or another?

This link has given me concern about getting tied to TenserFlow.


texts:
I have a dataset containing of only one column, using matplotlib I was able to do the scatter plot the following way

data = pd.read_csv
plt.scatter(data.index,data.coulumn1)

I want the same graph using seaborn but I am not sure how to implement the same approach in the following line

ax = sns.scatterplot(x="total_bill", y="tip", data=tips)


texts: If we use a generalized quadratic loss in a SVM model, what generalization performance bounds can be derived?

texts: If we write an objective function: $$\frac{1}{2}||\vec{w}||^2 + C \vec{\xi}' S \vec{\xi}$$ with the usual SVM constraints, and $S_{i,j} = e^{-\gamma || \vec{x_i} - \vec{x_i}||^2}$, where $\gamma$ is an hyper-parameter, what generalization bounds can be derived?


texts: Is NN with no hidden layer is behave like a regression?

texts:
Is a NN with no hidden layer is behave like a regression? What we could say that NN without hidden layer can say us?

If we have for instance 20 input and 4 output and I have no true label, is it similar to regression?

If it is a regression then it will be a supervised learning however there is no true label!

What is wrong inside this clarification?

more clarification:
Also, I have a dqn ( a deep learning structure inside q-learning approach)! Unfortunately I do not have the optimal answer of it! I am looking for a way to compare my results, Maybe I can compare by putting a un/supervised learning approach instead of DNN! For instance, you can put a DNN without hidden layer and as it behaves like the linear regression then you can put every supervised learning instead of it! As there is no True label I have problem with training section and estimate the DNN section with supervised learning! Also,as  it is MDP, I can not estimate the DNN section with unsupervised learning!


texts: Multivariate Polynomial Feature generation

texts:
I don't quite seem to understand the rules used to create the polynomial features when trying to find a polynomial model with Linear Regression in the multivariate setting.

Let's say I have a two predictor variables a and b. When generating polynomial features (for example using sklearn) I get 6 features for degree 2: y = bias + a + b + a * b + a^2 + b^2

This much I understand.
When I set the degree to 3 I get 10 features instead of my expected 8. I expected it to be this: y = bias + a + b + a * b + a^2 + b^2 + a^3 + b^3

What is the general formula of generating multivariate features? How does this look like in the 3rd degree?


texts:
I have been looking through research and cannot find too much on this topic. I believe that I might be searching for the wrong terms.

What I am trying to find is any literature on classifying a group of correlated and different images into a single class. Such that each image might have its own probability of being in a class but an overall meta probability is calculated for the entire group of images.

Almost as though additional images were used to either increase or lower the probability of belonging to a class.


texts: Topic modelling on only 24 documents gives the same "topic" for any K

texts:
Description:

I have 24 documents, each one of around 2.5K tokens. They are public speeches.

My text preprocessing pipeline is a generic one, including punctuation removal, expansion of English contractions, removal of stopwords and tokenization.

I have implemented and analyzed both Latent Dirichlet Allocation and Latent Semantic Analysis in Python and gensim. I am calculating the optimal number of topic by the topics' coherence.

Issue:

For any number of topics K (I have tried many, e.g. 10, 50, 100, 200) I always get the same combination of top words for all topics. Therefore, they are zero informative.

I have tried removing "useless" words by threshold the TF-IDF value, but still nothing.

Diagnostic:

Trying to understand what might be the cause, I used SVD on the TF-IDF matrix. My matrix is 24 x 8115, which leads to 24 singular values. This is the plot:

As you can see, there is no knee point.

Maybe I can't do this since I only have 24 documents?

Or am I ignoring something fundamental for topic modelling on such a small dataset?


texts: problem of entry format for a simple model in Keras

texts:
I apologize if this question is too elementary for this site. I am new in learning Keras and Tensorflow and I have the following type/shape problem on which I have already wasted too much time.

I entered this code (found on the web) to construct a keras model using sequential()

from keras.models import Sequential
from keras.layers import Dense, Activation

model = Sequential()
model.add(Dense(32, activation='relu', input_dim=100))
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])

I then want to try the function model.evaluate(). But I can't find in the documentation nor in my trials and errors under what format the entry of evaluate should be. Among many other things, I have tried:

import numpy as np
model.evaluate(np.random.random((100,)))

but I get a long error message ending in

ValueError: Error when checking input: expected dense_1_input to have shape (100,) but got array with shape (1,)

Anyone has an idea what is happening here? Just a simple line of code
creating a dummy entry that my model could evaluate() would unstuck me, I think.


texts: Application of machine learning in your job

texts:
For those of you who may implemented any machine learning application in your business context, can you share your experience and results obtained?

Amidst all the hype on AI I am interested to hear from actual applications.


texts: Faster RCNN is not able to recognize composed elements

texts:
I tried to fine tune Faster RCNN to realize object detection task with bounding boxes where I shall recognize face, eyes and mouth on image. However Faster RCNN seems to fail to recognize objects when they contain another one(face contains nose). I tried the same task on faces only and it worked good.
Is it a general behavior? How can I explain that? Is there a trick to get over it?


texts: How to retrain Glove Vectors on top of my own data?

texts: I am using GloVe and gensim for my project. I have a corpus of data (let's say mydata.txt) which has new words which are not in the existing Glove. So, how do I retrain glove so that the existing pre-trained glove must now include the new words on my corpus mydata.txt? I have been struggling and failed to find the solution for 2 weeks. The only similar post I found is this Improving existing GloVe Model


texts: How to detect overfitting of a stock screener

texts:
The project I am working on allows users to create Stock Screeners based on both technical and fundamental criteria.  Stock Screeners are then "backtested" by simulating the results of applying in over the last 10 years using Point-in-Time data. I get back the list of trades and overall graph of performance.
(If that is unclear, I have an overview here and there with more details).

Now a common problem is that users create overfitted stock screeners. I would love to give them a warning when the screen is likely to be over-fitted.

Fields I have to work with

All trades made by the Stock Screener

Stock, Start Date, Start Price, End Date, End Price

S&P 500 performance for the same time frame
Market Cap, Sector, and Industry of each Stock


texts:
I'm using a reviews data and trying to apply classifier model and get prediction. Here is the code i'm trying.
dataset = pd.read_csv('Scraping reviews.csv')
import numpy as np
X = np.linspace(0, 2*np.pi, 8)
y = np.sin(X) + np.random.normal(0, 0.4, 8)
X = X.reshape(-1, 1)

from sklearn.feature_extraction.text import CountVectorizer
count_vect = CountVectorizer()
X_train_counts = count_vect.fit_transform(dataset)
#X_train_counts=X_train_counts.reshape(4,1)
X_train_counts.shape
[out] (2,2)

from sklearn.feature_extraction.text import TfidfTransformer
tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts)
X_train_tf = tf_transformer.transform(X_train_counts)
#X_train_tf=X_train_tf.reshape(4,1)
X_train_tf.shape
[out] (2,2)

tfidf_transformer = TfidfTransformer()
X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)
#X_train_tfidf=X_train_tfidf.reshape(4,1)
X_train_tfidf.shape
[out] (2,2)

from sklearn.naive_bayes import MultinomialNB

clf = MultinomialNB().fit(X_train_tfidf, X_train_counts)

 [out] ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    <ipython-input-494-7734b71b758f> in <module>
          1 from sklearn.naive_bayes import MultinomialNB
          2
    ----> 3 clf = MultinomialNB().fit(X_train_tfidf, X_train_counts)

    ~\anaconda3\lib\site-packages\sklearn\naive_bayes.py in fit(self, X, y, sample_weight)
        613         self : object
        614         """
    --> 615         X, y = self._check_X_y(X, y)
        616         _, n_features = X.shape
        617         self.n_features_ = n_features

    ~\anaconda3\lib\site-packages\sklearn\naive_bayes.py in _check_X_y(self, X, y)
        478
        479     def _check_X_y(self, X, y):
    --> 480         return self._validate_data(X, y, accept_sparse='csr')
        481
        482     def _update_class_log_prior(self, class_prior=None):

    ~\anaconda3\lib\site-packages\sklearn\base.py in _validate_data(self, X, y, reset, validate_separately, **check_params)
        430                 y = check_array(y, **check_y_params)
        431             else:
    --> 432                 X, y = check_X_y(X, y, **check_params)
        433             out = X, y
        434

    ~\anaconda3\lib\site-packages\sklearn\utils\validation.py in inner_f(*args, **kwargs)
         70                           FutureWarning)
         71         kwargs.update({k: arg for k, arg in zip(sig.parameters, args)})
    ---> 72         return f(**kwargs)
         73     return inner_f
         74

    ~\anaconda3\lib\site-packages\sklearn\utils\validation.py in check_X_y(X, y, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, multi_output, ensure_min_samples, ensure_min_features, y_numeric, estimator)
        805                         ensure_2d=False, dtype=None)
        806     else:
    --> 807         y = column_or_1d(y, warn=True)
        808         _assert_all_finite(y)
        809     if y_numeric and y.dtype.kind == 'O':

    ~\anaconda3\lib\site-packages\sklearn\utils\validation.py in inner_f(*args, **kwargs)
         70                           FutureWarning)
         71         kwargs.update({k: arg for k, arg in zip(sig.parameters, args)})
    ---> 72         return f(**kwargs)
         73     return inner_f
         74

    ~\anaconda3\lib\site-packages\sklearn\utils\validation.py in column_or_1d(y, warn)
        843         return np.ravel(y)
        844
    --> 845     raise ValueError(
        846         "y should be a 1d array, "
        847         "got an array of shape {} instead.".format(shape))

    ValueError: y should be a 1d array, got an array of shape () instead.

    I tried reshaping X_train_counts, X_train_tf, X_train_tfidf but nothing is working. Please help me with this. Thanks.


texts: I just finished reading Delving Deep into Rectifiers paper. This paper proposes a new activation function called PReLU. Maybe it is obvious, because the paper did not mention it, but I want to know when is the parameter of a PReLU updated? Is it updated before weight update or after weight update?


texts: why would you mask out padded activations from the training loss?

texts:
I've followed taming-lstm for training a LSTM model on a NLP task in batches with various sentence lengths. One of his main points is:

Trick 3: Mask out network outputs we don’t want to consider in our loss function

and at least for my task, that was simply not true. I've tried different batch sizes, with heterogeneous sentence lengths in each batch, and reached the same model quality when I trained with the complex loss the author is suggesting and a simple cross-entropy loss. Is it just for my specific task and in other cases there's a sound theoretical/empirical evidence that this trick helps? It's certainly complicating the code...


texts: Do I load all files at once or one at a time?

texts:
I currently have $1700+$ CSV files. Each of them is in the same format and structure, give or take a row or possibly a column at the end. Each CSV is $\approx 3.8$ MB.
I need to perform a transformation on each file

Extract one data set, perform a summation and column select, then store inside a folder.
Extract an array, no need for column names here, then sore inside a folder.

From an algorithmic POV, is it better to load each file, perform the actions needed and then move on to the next file?
OR
Do I load all files, perform the action on all files and then store to hard drive?
I know how to do the actual process, I am after a 20,000 feet POV of dynamical programming/optimisation.
Thanks.