texts: Tensorflow model performs poorly on new data, don't know where to look at

texts:
I have a model that I trained over a training set. I use a simple 3 layer Dense model with 11 inputs (floats) and 3 outputs (floats). The input are normalised using tf.keras.utils.normalize. I am working with a regression problem.
I use a small subset of the training set also as testing set (using the validation_split). The model was adequate during training (RMSE of 70 and accuracy of 84%). When I evaluated the model on new data (but similar to the training data), the model performed really bad: first the loss error RMSE was 60 (lower than the one during training) and second the accuracy was dropped to 14%.
I know I am not very specific with my description, but I am looking for possible pointers to where to look at when there is such a huge difference in performance from training to real-data.
Here is roughly how I set up the model:
    x_train, y_train= self.get_training_data()
    x_train = tf.keras.utils.normalize(x_train, axis=1)

    model = tf.keras.models.Sequential([
      tf.keras.layers.Dense(50, activation=tf.nn.relu),
      tf.keras.layers.Dropout(0.4),
      tf.keras.layers.Dense(50, activation=tf.nn.relu),
      tf.keras.layers.Dropout(0.4),
      tf.keras.layers.Dense(50, activation=tf.nn.relu),
      tf.keras.layers.Dropout(0.4),
      tf.keras.layers.Dense(3)
    ])

    model.compile(optimizer='adam', loss=self._root_mean_squared_error, metrics=['accuracy'])

    es_callback = keras.callbacks.EarlyStopping(monitor='val_loss', patience=3)

    history = model.fit(x_train, y_train, validation_split = 0.3, epochs=100, batch_size=50, shuffle=True, callbacks=[es_callback])

Thanks!


texts: Accuracy stagnated while training notMNIST data

texts:
I am a beginner in machine learning. I have built a logistic classifier in Python using TensorFlow to train on notMNIST dataset. My code is as such:

weights = tf.Variable(tf.truncated_normal(shape = [784, 10]))
bias = tf.Variable(tf.zeros(shape = [10]))
logits = tf.matmul(features, weights) + bias
prediction = tf.nn.softmax(logits)
cross_entropy = -tf.reduce_sum(labels * tf.log(prediction), reduction_indices=1)
loss = tf.reduce_mean(cross_entropy)

train_feed_dict = {features: train_features, labels: train_labels}
valid_feed_dict = {features: valid_features, labels: valid_labels}
test_feed_dict = {features: test_features, labels: test_labels}

is_correct_prediction = tf.equal(tf.argmax(prediction, 1), tf.argmax(labels, 1))
accuracy = tf.reduce_mean(tf.cast(is_correct_prediction, tf.float32))

epochs = 5
batch_size = 50
learning_rate = 0.1

optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)

validation_accuracy = 0.0

with tf.Session() as session:

    session.run(tf.global_variables_initializer())
    batch_count = int(math.ceil(len(train_features)/batch_size))

    for epoch_i in range(epochs):

        for i in range(batch_count):

            session.run(optimizer, feed_dict = train_feed_dict)
            print(session.run(accuracy, feed_dict = train_feed_dict))

However, the problem is that while the training loss is decreasing continuously, the accuracy wavers initially, and then finally stagnates (at around 0.062). I am not able to understand what's wrong with the code. Any help would be appreciated. Thanks.


texts: Transitioning from a python script for data transformation to BigQuery

texts:
So I have a dataset spread over multiple and ever-growing excel files all of which looks like:

email
order_ID
order_date

user1@email.com
1234
23-Mar-2021

user2@email.com
1235
23-Mar-2021

user2@email.com
1236
23-Mar-2021

user1@email.com
1237
24-Mar-2021

user3@email.com
1238
28-Mar-2021

End goal is to have two distinct datasets as:
First one being Orders: (Public. For analysis, trading emails with user_IDs for anonymity and marking returning for further analyses)

user_ID
order_ID
order_date
is_returning?

1
1234
23-Mar-2021
0

2
1235
23-Mar-2021
0

2
1236
23-Mar-2021
1

1
1237
24-Mar-2021
1

3
1238
28-Mar-2021
0

Second one being: Users (Private for retaining users_info. Have other columns besides email as well but idea is the same)

user_ID
email

1
user1@email.com

2
user2@email.com

2
user2@email.com

1
user1@email.com

3
user3@email.com

In pandas, I have following procedure to go about:

Read all files:
 input_directory = 'Data/'
 files =  os.listdir(input_directory)
 files

Combine them:
 all_data = pd.DataFrame()
 for df in data_dict.values():
     all_data=pd.concat([all_data,df]).drop_duplicates().reset_index(drop=True)
 all_data = all_data.drop_duplicates()

Assign user IDs:
 all_data['user_id'] = all_data['email'].factorize()[0]

Assign returning flag:
 all_data['is_returning'] = all_data.user_id.duplicated().astype(int)

Pushing both to BQ:
 columns = #list of private columns
 all_data.drop(columns, axis = columns).to_gbq(#parameters, table = `public_dataset.orders`)
 all_data[columns].to_gbq(#parameters, table = `private_dataset.users_db`)

Since files are continually coming, I have to reassign user_IDs again and again. (Because searching a corresponding email for each row in customers db is way more expensive than simply reassigning IDs. All analyses are anonymous and based on total numbers so it wont hurt as well. Also mantaining a local DB is also not preferrable as data is incoming in an excel format) I can have a script to push data once a file is come for only that file but cannot because of

Not being able to emulate pd.factorize()
And dupicated() on BigQuery
Not knowing in advance what rows are already there since files may have duplicated records as well,

I have to do this again and again. So how can I transition this whole pipeline to BQ effectively and cost-efficiently? Any help regarding bettering the situation is highly appreciated


texts: Should predictive modelling response variable be measured in time after the predictors?

texts:
I am interested in modelling my dataset in a predictive way to predict the outcome (eg some event occurance) at some time in the future. I have been getting confused whether the outcome variable for each timepoint should therefore be chronologically in the future compared to the predictor variables measured for that timepoint row, so therefore the predictors relate to a future event as opposed to an event which has already occured. Therefore if the current dataset has outcome variable which relates to an event occuring within the last period of time, should this be changed so it relates to the consequent following period of time after the predictor variables were measured?
I hope this makes sense, hard to describe. I am still getting used to longditudinal analytics!


texts:
Im new to ML, I have a data set for Music sales info for Vinyls, the data set contains:

Author
Album Title
Genre
Country
RevenueGenerated
AverageRevenueGenerated

My goal is to create a Model which I can help me understand which Music may generate a lot of revenue (Boolean). I created a field AverageRevenueGenerated which is the average of all Revenue generated for all artists.
Im looking for a tool that can help me associate or generate insights based on input signals above. This cold be automatically or a specific guide that allows me to say for example if:

UK + Industrial
IT + Opera
Daft Punk + Electro

Will be potential high revenues.

I found house prices example: https://yalantis.com/blog/predictive-algorithm-for-house-price/ is it the same type of problem?  I'm looking which input signals may be the highest revenue. Any insights or pointers will be helpful.


texts:
This agnostic-model is not well addressed in research papers. I read articles where it was used to test the accuracy of the models, trying to understand the importance of individual features and their contribute to the model.
I saw values ranging from negative values to 10 or even more.
I am wondering which would be the expected values from a such method and which considerations should be done. I would expect that, after extracting many features from data and building new ones, there might correlation among factors, so for feature selection it would be considered for example a chi-squared or other statistical tests based on the feature type.
This would mean to not consider those variables that result highly correlated for model building, if I am right.
So what would be the purpose of running on the model testing a permutation based feature importance?
Again, many papers discuss about model performances in terms of ROC, AUC, Accuracy, f1-score, recall, precision, ... but they do not mention anything about the contribute of each feature on the model, especially when they add more features based on the selection.
And it is weird since for building a model you should be able to understand which factor has given more contribute. Would it be enough to use a statistical test like Chi-square?


texts: What is the point of tensors in CNNs? Why not simply reshape the data into matrices?

texts:
Take the following tensor:
$$
\left[\begin{array}{cc}
a & b & c\\
d & e & f\\
g & h & i\\
\end{array}\right]
$$
$$
\left[\begin{array}{cc}
j & k & l\\
o & n & m\\
p & q & r\\
\end{array}\right]
$$

Where each matrix represents a channel.

This could be reshaped fairly easily into a vector:
$$
[a,d,g,b,e,h,c,f,i,j,o,p,k,n,q,l,m,r]
$$

And then concatenated row-wise with other vectorized tensors to form a typical flat-file dataset of dimension $N \times P$, where $N$ is the number of training samples and $P$ is the product of the tensor dimensions.

Rather than futzing with a convolutional layer, one could simply constrain ones weights to be zero in the subsequent layer.

If $X$ is a flat $N\times P$ dataset of concatenated vectorized tensors, then the convolutional weights would form a sparse matrix, with the first two columns of a $P \times 4$ convolutional "layer" being $2\times 2\times 2$ filter being
$$
\left[\begin{array}{c}
w 0\\ w0\\ 00\\ ww\\ ww\\ 00\\ 0w\\0w\\00\\w0\\ w0\\00\\ww\\ww\\0w\\0w\\0w\\00
\end{array}\right]
$$

This seems to me more intuitive than the tensor formulation, and could be computed fairly simply using sparse matrix packages.  Perhaps it is partly matter of taste.  But I'm curious: is there anything special about the tensor paradigm -- either mathematically or computationally -- that is superior to the flattened representation?  I understand that computers convert matrix algebra to for-loops "under the hood", but doesn't the advent of the GPU make such explicit looping irrelevant?


texts:
Is there a way to run complex list comprehensions like the following on GPU?

[[x[index] if x[index]>len(x) else x[index]-1 for x in slice] if (len(slice)==1) else slice for slice,index in zip(slices,indices)]

To what degree is it Possible? Do I have to convert it to some kind of numpy comprehension (if so what part is speciffically possible/necessary)

The goal is performance optimization on large datalists/arrays.


texts: Pretrained model to classify text as address or not

texts:
I am training a text classifier for addresses such that if given sentence is an address or not.
Sentence examples :-
(1) Mirdiff City Centre, DUBAI United Arab Emirates
(2) Ultron Inc. <numb> Toledo Beach Rd #1189 La Salle, MI <numb>
(3) Avenger - HEAD OFFICE P.O. Box <numb> India

As addresses can be of n types it's very difficult to make such classifier. Is there any pre-trained model or database for the same or any other non ML way.


texts: Geospatial Social Network Analysis Visualization

texts:
I have a data set that keeps track of who referred someone to a program, and includes the geo coordinates of both parties for each record.

What will be the best way to visualize this kind of data set? This visualization should also be able to use the geo coordinates to place this entities in the map to form clusters, or to superimpose them on a real map.

I am interested in an algorithm and/or a library that will help me do this. Library should be preferably written in Java, Python, Scala, or NodeJS. The record count can be as big as a thousand or hundreds of thousands.


texts: How can I predict traffic based on previous time series data?

texts:
If I have a retail store and have a way to measure how many people enter my store every minute, and timestamp that data, how can I predict future foot traffic?

I have looked into machine learning algorithms, but I'm not sure which one to use. In my test data, a year over year trend is more accurate compared to other things I've tried, like KNN (with what I think are sensible parameters and distance function).

It almost seems like this could be similar to financial modeling, where you deal with time series data. Any ideas?


texts: How do I add a column to dataframe on condition that certain observations in my dataframe contain a target word in Python?

texts:
Here's what my data frame look like:

Number  Age      Famous_for
1       35       "businessman chairman of IBM (1973–1981)"
2       42       "musician (House of Freaks Gutterball)"
3       87       "baseball player (Oakland Athletics)"

I want to create additional column that will contain a dummy variable whether or not person was involved with entertaining business. Something like that:

Number..Age..................Famous_for.....................Entertaining
1.......35.......businessman chairman of IBM (1973–1981)........0
2.......42.......musician (House of Freaks Gutterball)..........1
3.......87.......baseball player (Oakland Athletics)............0

How do I create a column based on the certain words in the famous for column (such as "musician", "club", "actor" and etc)?
I tried the following:

df['entertaining'] = np.where(df['famous_for']>="musician", 1, 0)

but it wouldn't work.
How can I do this in Python?


texts:
In training session, model fitting happens to reduce error. But does KNN do this?

Reducing error only happens due to changing K value and number of features, isn't it?

So training set and test set is only for do things below, right?

Train model with training set
Given test sample, model finds K-nearest neighbors in training set
Do classification or regression for test sample
Find accuracy with MSE or RMSE


texts: A clear visualization of a two-way ANOVA

texts:
To provide a full yet simple picture of a 3-level, one-way ANOVA, I use the following visualization where variation within each group (the filled circles) and variation between the groups (black arrows) are simple to be understood.

But I'm wondering if it could be possible to extend the current visualization to a 2 x 3 two-way ANOVA (adding another way with two groups to the current visualization)?

(Note: the dashed vertical lines denote each group's mean)


texts: What is the output of multivariate LSTM model?

texts:
I am currently trying to build an LSTM model by using multivariate inputs, but I don't understand what exact output I am predicting.

I am currently using 5 features in the data i.e. 'Time', 'Avg CPU Load', 'F1', 'F2', 'F3' as input data.

X_data = data[['Time', 'Avg CPU Load', 'F1', 'F2', 'F3']]

My main goal is to predict 'Avg CPU load' by using the other features as an input data, but I am not sure if I am doing that.

In these features I have made 'Time' as an index in the dataframe.
Then I have used 60 timesteps in the below code.

new_X_train=[]
new_Y_train=[]
for i in range(len(X_data)-timesteps-1):
    t=[]
    for j in range(0,timesteps):

        t.append(X_data[[(i+j)], :])
    new_X_train.append(t)
    new_Y_train.append(X_data[i+ timesteps,1])

The variables 'new_X_train' and 'new_Y_train' have the inputs data that have to be inserted in the model. Then I define my model which is..

sq = Sequential()
sq.add(LSTM(units = 100, return_sequences = True,
           input_shape = (None, 4)))
sq.add(Dropout(0.2))

sq.add(LSTM(units = 100,return_sequences = True))
sq.add(Dropout(0.2))

sq.add(LSTM(units = 100,return_sequences = True))
sq.add(Dropout(0.2))

sq.add(LSTM(units = 100))
sq.add(Dropout(0.2))

sq.add(Dense(units=1)) # only one output

sq.compile(optimizer = 'adam', loss = 'mean_squared_error')

sq.fit(new_X_train, new_Y_train, epochs = 5, batch_size = 20)

Once the model is trained I load the test data and implement 60 timesteps to i.e I do the same for what I have done for the training data.
And then I use the X_test data to predict the values.

predicted = sq.predict(X_test)

Now I am not sure what output the above code is predicting, like what feature is it trying to predict?
And how to make my model predict 'Avg CPU load' feature?


texts: How to create a global model with personalized features for multi-label classification problem

texts:
I'm trying to predict additional recipients of a message given the content of the message (like subject and body) and the current recipients of the message.

for ex: I have 4 users in the system U1, U2, U3 and U4

I have the following messages between them

message-id  message-content sender  recipients
     m1          text        U1      U2,U3
     m2          text        U1      U2
     m3          text        U1      U3
     m4          text        U2      U3,U4
     m5          text        U3      U4

As you can see U1 only communicates (Send/Receive) with U2 and U3. Similarly, U2 communicates with U1, U3 and U4 while U3 communicates with U1, U2 and U4. Lastly U4 communicates with U2 and U3.

My goal is when a user is composing a message and already entered some content and recipients on the To line, I want to suggest other recipients the user should include in the To line. The suggestion should be relevant to the message's content and also only contain users with whom the user has communicated with previously.

So for U1 we should never recommend U4 as there is no communication between them before. Similarly, for U4 we would never suggest U1.

I'm currently thinking of solving this as a multi-label classification problem where I generate a personalized model per user using Binary Relevance method.

For U1, the data looks as follows

message-id  message-content U2  U3
   m1          text         1   1
   m2          text         1   0
   m3          text         0   1

Now, I can train a OneVsRest classifier (from scikit-learn) on this data using svm.LinearSVC (or some other classifier) to predict multi-recipients given a message. I'll use content and recipients of the message as features. This works.

The problem is this needs a personalized model per user. Given I have 10's of millions of users and each user sends/receives 1000's of messages per month and communicates with 1000's of other users overall, how can I create a single global model with personalized features for prediction?

I want to avoid creating millions of personal model per user if possible and want to train a single global model using data for all users and then do personalized predictions.

Is multi-label classification not suitable for learning a global model? Should I instead use a ranking model approach?

Please suggest how can I do this?


texts: Can my training and testing data be of different timesteps length?

texts:
I am trying to use a deep neural network for a regression problem. I have 50 samples of 600 timesteps each. I am planning to use 40 for training and 10 for validation.
If it matters, I have used a neural network with input layer, lstm, dense, dense(output).
After my network gets trained, can I use it to predict for a timeseries of a smaller length (200 timesteps)?

The input and output dimensions (i.e. features) will remain same, just the length (i.e. number of timesteps) is smaller


texts: LinearRegression with fixed slope parameter

texts:
I have some data $(x_{1},y_{1}), (x_{2},y_{2}), ..., (x_{n},y_{n})$, where both $x$ and $y$ represent real numbers (float). I want use Scikit-learns LinearRegression model to fit a model of the form:
$y_{i} = b_{0} + b_{1}x_i + e_{i} $
Typically, I know that OLS is used to compute the parameters $b_{0}, b_{1}$. However, in my case, I happen to know that  $b_{1}=c$ so I only want to fit $b_{0}$. Is there a way to force scikit-learn to use $b_{1}=c$ as the slope ratio and only estimate the intecept $b_0$, or is a custom class necessary?


texts: Simple implementation of 2D convolution neural network (CNN)

texts: I am looking for a good reference implementation (code) of 2D CNN (convolution neural network) using simple math operations instead of framework's high level tensor operation. This is for inference only. Many open source code implementations out are quite complex and utilize high level operations.


texts: R Studio - grepl compare a column in a dataframe to a list of pattern

texts:
I have a column named "MATCH" in a dataframe and a list of patterns named "PATTERN".

df1.MATCH <- c("ABC", "abc" ,"BCD")
df1 <- as.data.frame(df1.MATCH)
df2.PATTERN <- c("ABC", "abc", "ABC abc")

I want to use grepl to compare MATCH column with PATTERN, if true, I will apply my functions. The desired result would be "ABC" matches "ABC" and "ABC abc". This is the code I used:

df1 %>% filter(grepl(df1.MATCH,df2.PATTERN ))%>% ...

I get error:

"Warning message: In grepl(TXN_GROUP, parm[3]) :argument 'pattern' has length > 1 and only the first element will be used"

I understand I can't use grepl to a list of vectors. Is there any way to solve it?


texts: How to improve model accuracy by redistributing training data over test, validation, and training data subsets after training

texts:
How do I improve model accuracy by redistributing training data over test, validation, and training data subsets after training, and then training a new model with the updated data subsets.
The model has 60 inputs and 3 outputs. The total number of (balanced) exemplars is around 10k.
The data subset splits and the accuracies of the trained model on the subsets were as follows:
test subset 30% (accuracy 78%)
validation subset 21% (accuracy 80%)
training subset 49% (accuracy 99.8%)
Should I experiment with swapping half of the miss-classified exemplars from the validation and test subsets with an equal number of correctly classified exemplars in the training subset?


texts: Orange: Predictions widget on same data gives different results

texts: I have built a simple tree predictor and then applied this to a new set of data. This was giving a very different probability distribution so I tested connecting the same training data and comparing probability scores across the two predictions widgets in the following flow.The pre-processing steps are the same for both models (Group Mosaic + select columns), but the Predictions widgets give different outputs. Any idea why this might be? I'm on version 3.13.0.


texts: Sub topics with Latent Dirichlet Allocation

texts:
I'm training an LDA model with gensim's LdaMulticore. The topics look great, but knowing the domain I know there exists topics within topics but I'm not quite sure the best way to model this.

I've come across this implementation of Hierarchical LDA, but I'm having a hard time implementing it (no community support). And I don't think gensim's hdpModel is what I want, given this discussion.

I'm currently doing this:

1) train LDA model on all records to get general topics

2) use this LDA model to assign each record a primary topic

3) for each topic, retrieve only the records that were assigned that topic

4) train a new LDA model only on the filtered records (for example, where topic ID == 3) to generate sub topics from filtered set

5) Assign each record a general topic ID and a sub topic ID

Is this a valid way to get topics and sub topics? Should I be weary of this approach?

Thanks for any insight.


texts: Model for extracting common context from a similar cluster of sentence

texts:
I have multiple clusters of similar sentence embeddings (one cluster holding one type of similar sentences)
Is there any unsupervised model that can extract the common context from each cluster in:

a dense embedding or
in a sentence format altogether

I am gonna use BERT to evaluate the dense sentence embedding and apply hierarchical clustering to create the clusters (I haven't evaluated how well it is gonna work yet), then apply your suggested model to extract the common context.
P.S. I would be thankful if you could provide the model for the second option.


texts:
MOBA team games have teams composed of a subset of 5 heroes from a larger set of possible heroes (say 100 heroes in the larger set)
For example, a game can be between a team with heroes 1,8,43,65 and 71 and a team with heroes 3,7,23,41 and 45.
What is the best way to train a model that predicts the outcome based on team compositions?
For example, one option would be to have something like this:
|Label  | Hero1 | Hero2 | Hero3 | Hero4 | Hero5 | Hero6 | Hero7 | Hero8 | Hero9 | Hero10 |
|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|--------|
| 1     | 1     | 8     | 43    | 65    | 71    | 3     | 7     | 23    | 41    | 45     |

While another option could be instead of having 10 numeric columns for the hero selection, to have 200 boolean columns (100 for team A and 100 for team B) which would have a value of true if that hero was included in that team.
Which would be a better option? Or is there an even better option outside of those two?
Thanks!


texts: How to write out the definition of the value function for continous action and state space

texts:
In the book of Sutton and Barto (2018) Reinforcement Learning: An Introduction. The author defines the value function as.

$$v_{\pi}(\boldsymbol{s})=\mathbb{E}_{\boldsymbol{a}\,\sim\, \pi}\left[\sum_{k=0}^{\infty}\gamma^kR_{t+k+1}\,\bigg|\,\boldsymbol{s}_t=\boldsymbol{s} \right]$$

If $\boldsymbol{a}\in \mathcal{A}$ and $\boldsymbol{s}\in \mathcal{S}$ are continuous I would think by using Bellman's equation for the state-value function that this can be written as the integral

$$v_{\pi}(\boldsymbol{s})=\int_{\boldsymbol{a}\in\mathcal{A}}\pi\left(\boldsymbol{a}|\boldsymbol{s} \right)\int_{\boldsymbol{s}'\in \mathcal{S}}p(\boldsymbol{s}'|\boldsymbol{s},\boldsymbol{a})\left[R_{t+1}+\gamma v_{\pi}(\boldsymbol{s}')\right]d\boldsymbol{s'}d\boldsymbol{a}.$$

Is this correct?

Also without using Bellman's equation does the integral definition of the state-value function look like this?

$$v_{\pi}(\boldsymbol{s})=\int_{\boldsymbol{a}\in\mathcal{A}}\pi\left(\boldsymbol{a}|\boldsymbol{s} \right)\int_{\boldsymbol{s}'\in \mathcal{S}}p(\boldsymbol{s}'|\boldsymbol{s},\boldsymbol{a})\left[R_{t+1}+\gamma \left[\int_{\boldsymbol{a}'\in\mathcal{A}}\pi\left(\boldsymbol{a}'|\boldsymbol{s}' \right)\int_{\boldsymbol{s}''\in \mathcal{S}}p(\boldsymbol{s}''|\boldsymbol{s}',\boldsymbol{a}')\left[R_{t+2}+\gamma\left[\cdots\right] \right]d\boldsymbol{s''}d\boldsymbol{a}'\right] \right]d\boldsymbol{s'}d\boldsymbol{a}$$

Are my integral versions correct?


texts: Train test leakage doubt for Time series

texts:
I have a very silly doubt about potential leakage of information during train test splitting. My dataset is a timeseries of multiple features for the year of 2018 where every row are observations taken at every 3 second timestamp. The data goes into a function which filters the data between a start,end date range and produces some grouped aggregations df.groupby(col).agg('mean','meadian'....)where colis a categorical var. The start,end date ranges are completely exclusive and dont overlap for ex : 2018-01-01:2018-15-01,2018-16-01:2018-30-01...and so on. So the windows produced are not overlapping at all.
I save these batches on the disk and load all of them into a single dataframe and then do a random train test split. Do you think this way I am leaking some info in the test set ?
Moreover I want to scale my features before producing the aggregations. So there has to be a fitted scaler at the very first step.
I came up with two approaches that I could follow to ensure the process error prone.
First:

1. Split the dataset by time order into train test
2. Apply scaler on train and transform on test
3. Then generate aggregated data for both train test
4. train on aggregated train predict on aggregated test
5. For a new test set , transform it using fitted scaler, generate the aggregated test for and predict on it

Second:
1. take the whole dataset and for every timeframe window of 15 days apply scaler fit method and transform
2. produce the aggregations on the 15 days batched data.
3. write the batches to disk and load them all into a dataframe and simply do a random train test split.
4. for a new test set, make timeframe windows of 15 days apply scaler fit and transform on the test and then produce aggregations  and then simply predict on it

Which of the two approaches i should try ? If I consider the first approach should I do it for multiple time folds , like first 1-5 months train then next 6-7 months test, 2-6 months train and 7-8 months test and so on ?


texts: Information leakage when train/test are truly i.i.d.?

texts:
I am well aware that to avoid information leakage, it is recommended to fit any transformation (e.g., standardization or imputation based on the median value) on the training dataset and applying it to the test datasets. However. I am not clear what is the risk of applying these transformations to the entire dataset prior to train/test split if the the data is iid and the train/test split is indeed random?
For example, if the original data set has certain statistical characteristics(e.g., mean, median, and std) then I would expect a random data spilt with generate a train and test datasets that have the same statistical characteristics. Therefore, standardizing the entire datasets and then  splitting should produce the same results as splitting the dataset, standardizing based on the train database and transforming the test dataset. The same argument can be made for imputation based on the median value.
Am I missing something?


texts: Name of this type of cross variable interaction Plot

texts:
In recent papers of Social Network Analysis , I have been observing this particular type of graph used increasingly. I looked for what they are called to replicate the graph, but couldn't find the name of it?

Plot Reference is here:

Bots increase exposure to negative and
inflammatory content in online social systems


texts: Having trouble reducing MSE error for SVR model sklearn

texts:
I'm trying to create an SVR model to predict the number of comments a headline will receive for the following dataset :
https://www.kaggle.com/benjaminawd/new-york-times-articles-comments-2020?select=nyt-articles-2020.csv
I tried a few nlp techniques for creating feature vectors for my training + test sets including stop-word removal, n-grams, and word-stemming (from a tutorial on pluralsight.com). I felt it was best to use an SVR model in sklearn, but my MSE comes out to be >500 (the number of comments range is between 0 and 5,000). I had previously looked up this post for some tips as well:
Large mean squared error in sklearn regressors
and thought of removing small chunk of outliers (i.e. those entries with a very high number of comments) from the dataset, but was a little doubtful because I read that SVR was robust to outliers, at least according to this post:
https://towardsdatascience.com/unlocking-the-true-power-of-support-vector-regression-847fd123a4a0
I also changed the kernel parameter for SVR, but that didn’t really change the MSE much either.
Does anyone have suggestions as to what I can do to reduce the MSE and RMSE as much as possible?
Below is a snap of my code in case you want to review it (modified from a tutorial on pluralsight.com):
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

from sklearn.feature_extraction.text import HashingVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.datasets import load_iris
from sklearn.linear_model import LinearRegression
from sklearn.svm import SVR
from sklearn.feature_extraction import text
from nltk.stem.snowball import SnowballStemmer
from sklearn.naive_bayes import GaussianNB

from nltk.tokenize import word_tokenize
from nltk.probability import FreqDist

import pandas as pd

def stemmed_words(doc):
    return (stemmer.stem(w) for w in analyzer(doc))

def summarize_classification(y_test, y_pred):
    rmse = mean_squared_error(y_test, y_pred, squared=False)
    mse = mean_squared_error(y_test, y_pred, squared=True)

    print("Length of testing data: ", len(y_test))
    print("root mean squared error: ", rmse)
    print("mean squared error: ", mse)

stemmer = SnowballStemmer('english')
analyzer = HashingVectorizer().build_analyzer()

df = pd.read_csv('./archive/nyt-articles-2020.csv', encoding="ISO-8859-1")

X = df['headline']
Y = df['n_comments']

# frequency filtering
tokens = word_tokenize("\n".join(X.values))
freq = FreqDist(tokens)
frequent_words = []

for key, value in freq.items():
    if value >= 100:
        frequent_words.append(key.lower())

stop_words = text.ENGLISH_STOP_WORDS.union(frequent_words)

#stem_vectorizer = HashingVectorizer(n_features=2**10, norm='l2', analyzer=stemmed_words, ngram_range=(2,5)) #34 acc count
stem_vectorizer = HashingVectorizer(n_features=2**10, norm='l2',ngram_range=(2,3), stop_words=stop_words, analyzer=stemmed_words) #38 acc count

feature_vector = stem_vectorizer.transform(X)

feature_vector.shape

X_dense = feature_vector.todense()

X_dense.shape

x_train, x_test, y_train, y_test = train_test_split(X_dense, Y, test_size = 0.2)

x_train.shape, x_test.shape

y_train.shape, y_test.shape

clf = SVR(kernel='sigmoid').fit(x_train, y_train)

y_pred = clf.predict(x_test)
y_pred

print(y_test)
print(y_pred)

summarize_classification(y_test, y_pred)

Here's part of the output too:
Length of testing data:  3358
root mean squared error:  530.9297286668819
mean squared error:  281886.37678228883


texts: Importance of feature selection for boosting methods

texts:
While it is obviously clear that features can be ranked on basis of importance and many machine learning books give examples of random forests on how to do so, its not very clear on which occasions one should do so.

In particular, for boosting methods, is there any reason why one should do feature selection. Wouldn't the boosting methods themselves eliminate the low importance feature.

Isn't is just always better to add more features (if one didn't have the practical problem of time limitations).


texts:
I have two datasets. One is actual percentage of white population in counties in an american state and the other is the simulated percentage of white population in counties in an american state.

Bits about my simulation:

  It is a random simulation done on California map with two different
  agents, white and minority. Their total population is based on the
  real white to minority ratio in California. For example if there is
  70% white and 30% minority in California then the agents(say total
  100) would be 70 white and 30 minority. First the map is randomly
  populated with both the agents and then around 100 iteration of the
  simulation is performed. On every iteration agent moves based on
  certain conditions. Data is taken after the 100th iteration and it
  includes what percentage of the white and minority are in a certain
  county in California.

So below is the data from the state of California

california_actual_white = [0.52, 0.72, 0.9, 0.86, 0.91, 0.91, 0.67, 0.79, 0.89, 0.77, 0.89, 0.84, 0.9, 0.81, 0.82, 0.81, 0.87, 0.82, 0.71, 0.86, 0.86, 0.9, 0.86, 0.82, 0.89, 0.91, 0.82, 0.84, 0.93, 0.72, 0.85, 0.91, 0.8, 0.64, 0.88, 0.77, 0.76, 0.54, 0.67, 0.89, 0.61, 0.85, 0.55, 0.87, 0.88, 0.94, 0.87, 0.61, 0.87, 0.83, 0.73, 0.9, 0.88, 0.88, 0.9, 0.84, 0.75, 0.79]

california_simulated_white = [0.48, 0.54, 0.6, 0.62, 0.66, 0.69, 0.71, 0.71, 0.71, 0.72, 0.74, 0.75, 0.77, 0.78, 0.79, 0.79, 0.8, 0.8, 0.8, 0.81, 0.81, 0.82, 0.82, 0.82, 0.83, 0.84, 0.85, 0.85, 0.87, 0.87, 0.87, 0.88, 0.91, 0.92, 0.93, 0.93, 0.94, 0.94, 0.94, 0.94, 0.95, 0.95, 0.97, 0.97, 0.98, 0.98, 0.98, 0.98, 0.99, 0.99, 0.99, 0.99, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]

How would I found a metric of similarity between these two datasets.

I found out a these three option can be used to find similarity and also all of them have a method in Python:

  1) Earth mover's distance

  2) Kullback–Leibler divergence

  3) Cosine Similarity

But I have some doubts using these methods. They are

  1) With Kullback-Leibler divergence and Cosine Similarity, the value
  changes if i reshuffle both the arrays and compute both the metrics
  again but with Earth movers distance its not the case. It will give
  you the same value for the two datasets regarding the
  reshuffling/position of the data points which made me think to use
  this metric.

  2) But the second doubt is that you can use the K-L divergence and
  Earth movers distance only with two probability distribution. So i am
  not sure here that if the two datasets above are the probability
  distribution or not.

So my final two questions are -

1) Are the above datasets considered as probability distribution? If yes why?

2) If 1) is yes then what is the best method to determine the similarity. If 1) is no then what is the best method to determine the similarity ?

Before I was leaning towards Earth Movers Distance as I stated the reason in the doubts section.


texts: In Tensorflow, What kind of neural network should I use?

texts:
I am doing Tensorflow tutorial, getting what TF is. But I am confused about what neural network should I use in my work. I am looking at Single Layer Neural Network, CNN, RNN, and LSTM RNN.

-----------------------What I'm going to do is...-----------------------

There is a sensor which measures something and represents the result in 2 boolean ways. Here, they are Blue and Red, like this:

the sensor gives result values every 5minutes. If we pile up the values for each color, we can see some patterns:

number inside each circle represents the sequence of result values given from sensor.
(for example, 107 was given right after 106) when you see from 122 to 138, you can see decalcomanie-like pattern.

I want to predict the next result value, before the sensor imparts the result, with probability.
Machine has to know what the next will be, based on patterns from past results.

I may do supervised learning using past results. But I'm not sure which neural network or method is suitable. Thinking that this work needs pattern using past results (have to see context), and memorize past results, maybe LSTM RNN (long-short term memory recurrent neural network) would be suitable one.

Could you tell me which one is suitable for this work?


texts: Performance of UNet on low resolution images

texts:
I read the U-Net paper yesterday and it talked mostly about how it uses a small dataset (<100) of high-resolution images and performs efficiently.

I was wondering how it might perform on larger datasets (>1000) of low-resolution images.
One of it's properties is heavy use of Data Augmentation to compensate the size of training set as mentioned in the paper. Would that be as effective on low-res images?


texts: Detecting hand-written objects from images

texts:
Problem Statement: Identify the sub-region in an image with hand-written text/scribbling
(Attaching an example for reference. Masked some text for privacy)

Once I get the pixel values, I want to replace them with white pixels, so I remove the scribblings completely from the image.

(PS: I don't have a lot of training images to train a CNN. So, asking here in case the problem can be solved via a classical CV approach)


texts: How to do stepwise regression using sklearn?

texts:
I could not find a way to stepwise regression in scikit learn. I have checked all other posts on Stack Exchange on this topic. Answers to all of them suggests using f_regression.

But f_regression does not do stepwise regression but only give F-score and pvalues corresponding to each of the regressors, which is only the first step in stepwise regression.

What to do after 1st regressors with the best f-score is chosen?


texts: What is the difference between "Document Layout Analysis" / "Document Understanding" / "Document Structure Analysis"

texts:
All three terms sound super similar:

[...] document layout analysis is the process of identifying and categorizing the regions of interest in the scanned image of a text document. A reading system requires the segmentation of text zones from non-textual ones and the arrangement in their correct reading order. Detection and labeling of the different zones (or blocks) as text body, illustrations, math symbols, and tables embedded in a document is called geometric layout analysis.[2] But text zones play different logical roles inside the document (titles, captions, footnotes, etc.) and this kind of semantic labeling is the scope of the logical layout analysis.
Document layout analysis is the union of geometric and logical labeling.

Source: Wikipedia

The process of document structure and layout analysis tries to decompose
a given document image into its component regions and understand their functional
roles and relationships. The processing is carried out in multiple steps, such as preprocessing, page decomposition, structure understanding, etc

Source: Document Structure and Layout Analysis by Anoop M. Namboodiri
From this I would say that wikipedia calls it "geometric layout analysis" what Namboodiri calls "layout analysis". Wikipedia calls "logical layout analysis" what Namboodiri calls "document structure analysis".
Namboodiri uses "document understanding" as a broader term.
Do I understand this right?
(Is Document Understanding a sub-field of Information Extraction?)


texts: City pollution data for data mining project

texts:
I want to do my MSc Project on Data Mining. And I want to work on pollution in cities.
Are there accessible and useable pollution data (any type of pollution. e.g. air, water or forest pollution) in specific city or area?
If not, do you have any advice on how to get or search for them?
Thank you..


texts: Can embeddings generated by word2vec be similar for words which never share any words in the same sentence?

texts:
Is it possible for word2vec to produce similar embedding vectors for two words which never share any common words in the sentences that the words are found in?

Specifically, imagine I have the words A and B. Next imagine that I have words(A) and words(B), which represent the set of all words which show up in the same sentences as A and B, respectively. If the intersection of words(A) and words(B) is the null set (meaning the two words never have common words), then would it be possible for word2vec to put the embedding vectors for A and B in similar regions of the vectorspace?


texts: Training Dataset required for Classifier

texts: I am currently trying to develop a classifier in python using Naive Bayes technique. I need a dataset so that I can train it. My classifier would classify a new document given to it into one of the four categories : Science and technology, Sports, politics, Entertainment. Can anybody please help me find a dataset for this. I've been stuck on this problem for quite some time now. Any help would be greatly appreciated.


texts: Cyclic dependency between feature and predictor class

texts:
I have a feature which has specific categorical values ex(Technology, Hardware, Software, Marketing, Evnts etc). Based on this and some other features, I am trying to classify the dataset into 2 categories IsSoftwareSystem or NotSoftwareSystem. In this case is this cause a reduce in accuracy because i am feeding the category itself in the data and trying to predict the same.
Using Random Forest/XGB.


texts: How to compare two unsupervised anomaly detection algorithms on the same data-set?

texts: I want to solve an anomaly detection problem on an unlabeled data-set. The only information about this problem is that the anomalies population is lower than 0.1%. It should be notice that the size of the feature vector for each sample is 40. Is there any clear way to compare the performance of unsupervised algorithms?


texts: Unable to generate error bars with seaborn


texts: Knowing Joint probability distribution between feature-label space

texts:
I am doing a course CORNELL CS4780 "Machine Learning for Intelligent Systems". you can find the link here for the one I am going to refer 1st lecture

The professor explains, we have a sample

  $D ={ (X_1,y_1),(X_2,y_2), \ldots,(X_n,y_n)} \sim P$
      Where, (Xi,yi) is a feature-label pair. There is a joint distribution
      over the feature-label space and is denoted by $P$.

We never have access to the $P$, Only God knows $P$.  What we want to do in this supervised learning  task is to take data from this distribution and learn a mapping/function form $X$ to $y$.

I agree/understand till this point.

Then, Professor goes on to make a statement in the lecture, precisely at 34 minutes 26 seconds, that

"IF we had access to this distribution, everything would be easy". But he doesnt explain this statement.

Now my question is What would have been easy if we knew about the distribution ? Does he mean, if we had access to the distribution then we would know the probabilities of each of $(X_i,Y_i)$ pair. Then we can learn a mapping/parameters such that we reduce out of sample error?


texts: Loading file into and out of HDFS via system call/cmd line vs using libhdfs

texts:
I am trying to implement a simple C/C++ program for the HDFS file system like word count, it takes a file from the input path puts it into HDFS (where it gets split), processed my map-reduce function and gives an output file which I place back to the local file system.

My question is what makes better design choice to load the files into HDFS: From a C program call bin/hdfs dfs -put ../inputFile /someDirectory or make use of libhdfs?


texts: Feature extraction from resume using Python without rule based logic

texts:
I am working on a resume parser project. Currently, I am using rule-based regex to extract features like University, Experience, Large Companies, etc.

So basically I have a set of universities' names in a CSV, and if the resume contains one of them then I am extracting that as University Name.  In the same way I have a list of Large Companies in CSV and if the resume contains any of them then I flag it as Yes.

So these are rule-based logic and can never be fool-proof considering different countries have different resume formats.
Is there any other way of doing it to improve the accuracy and make it a global solution?


texts: Basic DNN with highly imbalanced dataset -- network predicts same labels

texts:
I will try to explain my issue at a high level, and I hope I'd be able to get some better understanding of the ML behind it all.
I am working with aggregated features extracted from audio files, so each feature vector is of size (1xN). The output would be a single sentiment label, Positive, Neutral, or Negative. I mapped these to 2, 1, 0 respectively (the labels are discrete by design, but maybe I could make it continuous?)

The dataset I am using is 90% neutral, 6% negative, and 4% positive, and I split these into train/dev/test. I wrote up a basic DNN in PyTorch, and have been training using CrossEntropyLoss and SGD (with nesterov momentum). The issue I am running into is that the network, after seeing only ~10% of the data, starts to predict only netural labels. The class weights converge to something like

tensor([[-0.9255],
        [ 1.9352],
        [-1.1473]])

no matter what 1xN feature vectors you feed in. I would appreciate guidance on how to address this issue.
For reference, the architecture is

DNNModel(
  (in_layer): Linear(in_features=89, out_features=1024, bias=True)
  (fcs): Sequential(
    (0): Linear(in_features=1024, out_features=512, bias=True)
    (1): Linear(in_features=512, out_features=256, bias=True)
    (2): Linear(in_features=256, out_features=128, bias=True)
  )
  (out_layer): Sequential(
    (0): SequenceWise (
    Linear(in_features=128, out_features=3, bias=True))
  )
)

def forward(self, x):
    x =  F.relu(self.in_layer(x))
    for fc in self.fcs:
        x = F.relu(fc(x))
    x = self.out_layer(x)
    return x

Not sure if NN actually makes sense -- could it be the relus between each hidden layer or the bias? Or something else?

Reposted from Stack Overflow here, since this forum is more appropriate: link


texts: Mathematical physics applications in present-day image processing

texts:
During the past few years several important areas of image processing and image classification or generation became dominated by convolutional neural networks.

I'm interested if there are any methods coming from mathematical physics context (like methods designed for solving ill-posed problems, spectral analysis, image deblurring and deringing), that outperform neural network-based approaches in 2017 for some common computer vision or image processing problem. Or methods that don't have any neural network-based rivals. Maybe in the field of biomedical image analysis (just a guess)?

The deeper and more specific the answer is, the better.


texts: Can I test my already trained model using the data it was trained on?

texts:
I have a model that predicts multiple choice answers to questions. I used an 80/20 train test split of my questions and tuned it.

The questions actually form part of a game aka 10 questions in a game. The data when split was randomly shuffled and split so the questions no longer form part of a game.

Can I now use the same questions it was trained and tested on to re-test all the questions but in order to determine the percentage of games won? Or when initially training my model should the train and test data be split by games instead of questions?


texts: How to handle highly Imabalanced classification?

texts:
I have been dealing with a classification problem. Real issue is the imbalance here
I have ~500,000 -ve samples and ~300 +ve samples.End result is predicted probabilities NOT hard 0-1 classification  Personally I am not a big fan of oversampling/under-sampling since they mess the distribution up. I also tried stratified sampling so that same proportion is kept while training but its not working. For my present approach I take a sample of -ve points (say 100,000) and then train with 300 +ve samples and getting an ROC-AUC of ~85 but almost no case predicted reaches probability >51% (after using model.predict_proba). Any tips on handling such an extreme imbalance? additionally I would only like to stay with tree based or boosting models since they provide interpretability (hence avoiding neural network, anomaly detection , autoencoder for now).
Any hints, resources appreciated!
Below is the code I am using.
df = read_data(path)
Pkl_Filename = path + "/catboost.pkl"
# if not os.path.exists(Pkl_Filename):
X = df.drop(ep_config['target_field'], axis=1)
y = df[ep_config['target_field']]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, stratify=y)
clf = CatBoostClassifier(eval_metric='Recall', use_best_model=True, scale_pos_weight=...)
sm = SMOTE(random_state=27, ratio=1.0)
X_train, y_train = sm.fit_sample(X_train, y_train)
smote = clf.fit(X_train, y_train, eval_set=(X_test, y_test))

# smote_pred = smote.predict(X_test)

# Checking accuracy
# accuracy_score(y_test, smote_pred)
# clf.fit(X_train, y_train, eval_set=(X_test, y_test))
with open(Pkl_Filename, 'wb') as file:
    pickle.dump(smote, file)
# else:
#     clf = pickle.loads(Pkl_Filename)
logging.info('the test roc is :{:.6f}'.format(roc_auc_score(y_test, smote.predict_proba(X_test)[:,1])))


texts: When I try to predict with my model I get an Attribute error

texts:
After I've created my model using keras sequential, I tried to start predicting on a small sample to see if it would work however I get this error and I have no idea why.

  error : AttributeError Traceback (most recent call last)
   in () ----> 1 example_results
  = model.predict(example_batch) AttributeError: 'NoneType' object has no attribute 'predict'

The code for my model can be found here :

https://colab.research.google.com/drive/1mxRIx5ufVsA0ljdTpL0Ud0qm2y39eyGX

I've done it using this method before but it refuses to work this time around.


texts: NLP Bert model to to calculate text similarity, same sentence but not close similarity

texts:
Dear expert here:
I have a simple program to calculate text similarity. The program is copied from internet. Initially, I have a list of sentences or stored in db and fetched from db, then

I make the list vectorizered and convered into bert model's vector, then using cosine to calculate the similarity.
2nd time, I instantiate a new vectorizer and vectorizer only one sentence and then compare against the first time bert model's vector, the result is different from 1st one.
3rd time, I using original vectorizer and vectorizered same sentence again, I notice the result is still different.

Conclusion，one has to vectorizer sentence one by one, not a list. Why???
from scipy import spatial
from sent2vec.vectorizer import Vectorizer

sentences = [
    "This is an awesome book to learn NLP.",
    "This is an awesome book to learn NLP.",
    "This is a great book to learn NLP",
    "This is a great book to learn NLP bert model",
    "DistilBERT is an amazing NLP model.",
    "We can interchangeably use embedding, encoding, or vectorizing.",
    "How about a chinese dinner? this shall be close to 1 or close to 0, I suppose it is close to 1, i.e. the distance is very big"
]

vectorizer = Vectorizer()
vectorizer.bert(sentences)
vectors_bert = vectorizer.vectors

dist_1 = spatial.distance.cosine(vectors_bert[0], vectors_bert[1])
dist_2 = spatial.distance.cosine(vectors_bert[0], vectors_bert[2])
dist_3 = spatial.distance.cosine(vectors_bert[0], vectors_bert[3])
dist_4 = spatial.distance.cosine(vectors_bert[0], vectors_bert[4])
dist_5 = spatial.distance.cosine(vectors_bert[0], vectors_bert[5])
dist_6 = spatial.distance.cosine(vectors_bert[0], vectors_bert[6])
print('dist_1: {0}, dist_2: {1}, dist_3: {2}, dist_4: {3}, dist_5: {4}, dist_6: {5}'.format(dist_1, dist_2, dist_3, dist_4, dist_5, dist_6))

test2 = []
test2.append(sentences[0])
vectorizer2 = Vectorizer()
vectorizer2.bert(test2)
vectors_bert2 = vectorizer2.vectors

dist_same_string_different_vectorizer = spatial.distance.cosine(vectors_bert[0], vectors_bert2[0])
print("dist same string different vectorizer is {0}".format(dist_same_string_different_vectorizer))

test3 = []
test3.append(sentences[0])
vectorizer.bert(test3)
vectors_bert3 = vectorizer.vectors
print("length of vector_bert3 is {0}".format(len(vectors_bert3)))
dist_same_string_same_vectorizer_different_time = spatial.distance.cosine(vectors_bert[0], vectors_bert3[0])
print("dist same string same vectorizer different time is {0}".format(dist_same_string_same_vectorizer_different_time))

/2/result
dist_1: 0.0, dist_2: 0.039356231689453125, dist_3: 0.06737476587295532, dist_4: 0.046457767486572266, dist_5: 0.12459474802017212, dist_6: 0.1990041732788086
dist same string different vectorizer is 0.21568220853805542
length of vector_bert3 is 1
dist same string same vectorizer different time is 0.21568220853805542

Process finished with exit code 0


texts: Implement the following loss function without interrupting the gradient chain registered by the gradient tape

texts:
I have spent five days trying to implement the following algorithm as a loss function to use it in my neural network, but it has been impossible for me. Impossible because, when I have finally implemented, I get the error:

  No gradients provided for any variable: ['conv1_1/kernel:0',
  'conv1_1/bias:0', 'conv1_2/kernel:0'

I'm implementing a semantic segmentation network to identify brain tumours. The network is always returning very good accuracy, about 94.5%, but when I have plotted the real mask with network outputs, I see that the accuracy is 0% because it doesn't plot any white dot:

By the way, the mask image only has values between 0.0, black, and 1.0, white, values.

So, I have decided to implement my own loss function, which has to do the following:

In a nutshell, sum mask image to output and count how many 2.0 values are in this sum. Compare this count of 2.0 values with the count of 1.0 values in the mask image, and get the error.

More detailed:

Converts the output's value of the model into 0.0 or 1.0.
Sum that output to the mask (which values are also 0.0 or 1.0).
Counts how many 2.0 are this sum.
Counts how many 1.0 are in the mask.
Returns the different between them.

My question is:

Is there a already tensorflow function that do that?

Now, I get that network output because I'm using Euclidean Distance. I have implemented the loss function using tf.norm:

def loss(model, x, y):
  global output
  output = model(x)
  return tf.norm(y - output)

And then, I use this loss function to tape gradients:

def grad(model, inputs, targets):
  with tf.GradientTape() as tape:
      loss_value = loss(model, inputs, targets)
  return loss_value, tape.gradient(loss_value, model.trainable_variables)

Maybe, what I am trying to do is another kind of distance.


texts:
I have received some NGS data ( sample below ) as a tsv file and shall analyze it in R.

# Metainfo 1
# Metainfo 2
        Sample: 8   9   10
        treatement  t1  t2  t1
probeset    symbol  feature ABC_001 ABC_002 ABC_003
1   a   11  9   118 151
2   b   12  237 770 710
3   c   13  221 966 390
4   d   14  609 723 376

In Bioconductor I read about 'SummarizedExperiment' and saw that it has a similar data structure as my tsv file (sample, feature, assay). Is there a way to load my tsv file directly as SummarizedExperiment?


texts: Max/Min of neurons at the hidden layer of a neural network

texts:
We have a neural network with inputs $x_1,...,x_n$, a single hidden layer with neurons $y_1,\ldots,y_m$, and outputs $z_1,...,z_v$. There are no activation functions (i.e., the activation functions are $f(x)=x$). We know that input is between -1 and 1, and we need the outputs to be between -1 and 1 as well. The network has been trained so the weights are all available. Given the weights, find the maximum and minimum values that each $y_i$ can take.

My effort: I thought of a simpler version of it. Let's say we have 2 inputs, 1 node at hidden layer, and one output. Then:

$$y_1=w_1 x_1 + w_2 x_2 $$
$$z_1=w_1' y_1$$

The maximum for $y_1$ occurs when $x_1=\mathrm{sign}(w_1)$ and $x_2=\mathrm{sign}(w_2)$. So, $y_1 \leq |w_1| + |w_2|$. On the other hand, $-1<y_1 w_1'<1$, so $y_1<1/|w_1'|$. Thus, $y_1\leq \min\{|w_1|+|w_2|,1/|w_1'|\}$. Similary, we can show that $y_1\geq - \min\{|w_1|+|w_2|,1/|w_1'|\}$.

Any idea for the general problem? Has this already been addressed in the literature?


texts: how to run bert's pretrained model word embeddings faster?

texts:
I'm trying to get word embeddings for clinical data using microsoft/pubmedbert.
I have 3.6 million text rows. Converting texts to vectors for 10k rows takes around 30 minutes. So for 3.6 million rows, it would take around - 180 hours(8days approx).

Is there any method where I can speed up the process?

My code -
from transformers import AutoTokenizer
from transformers import pipeline
model_name = "microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext"
tokenizer = AutoTokenizer.from_pretrained(model_name)
classifier = pipeline('feature-extraction',model=model_name, tokenizer=tokenizer)

def lambda_func(row):
    tokens = tokenizer(row['notetext'])
    if len(tokens['input_ids'])>512:
        tokens = re.split(r'\b', row['notetext'])
        tokens= [t for t in tokens if len(t) > 0 ]
        row['notetext'] = ''.join(tokens[:512])
    row['vectors'] = classifier(row['notetext'])[0][0]
    return row

def process(progress_notes):
    progress_notes = progress_notes.apply(lambda_func, axis=1)
    return progress_notes

progress_notes = process(progress_notes)
vectors_2d = np.reshape(progress_notes['vectors'].to_list(), (vectors_length, vectors_breadth))
vectors_df = pd.DataFrame(vectors_2d)

My progress_notes dataframe looks like -
progress_notes = pd.DataFrame({'id':[1,2,3],'progressnotetype':['Nursing Note', 'Nursing Note', 'Administration Note'], 'notetext': ['Patient\'s skin is grossly intact with exception of skin tear to r inner elbow and r lateral lower leg','Patient with history of Afib with RVR. Patient is incontinent of bowel and bladder.','Give 2 tablet by mouth every 4 hours as needed for Mild to moderate Pain Not to exceed 3 grams in 24 hours']})

Note - 1) I'm running the code on aws ec2 instance r5.8x large(32 CPUs) - I tried using multiprocessing but the code goes into a deadlock because bert takes all my cpu cores.


texts: Gym action space for board game with reward function

texts:
Im trying to design an openai gym environment that plays a quite simple board game where each player has 16 pieces that are exactly the same in regard to how they can move.

The board is 10x10 and each piece can go UP, DOWN, LEFT, RIGHT, UP_LEFT, UP_RIGHT, DOWN_LEFT, DOWN_RIGHT. They can move in that direction as many fields as pieces are in that line, including the piece that moves. So if I want to go LEFT I count all other pieces to my left AND my right add 1 for myself and then go that many fields to the left. The field may be obstructed though in which case the move is not possible.

So my question is:
How could I implement an action space for this? Would be discrete with the (sice of the board) * (how many actions[Up, DOWN...]) suitable? And how can I teach the rl AI (PPO2) that a move is not possible? Should I just give a negative reward and give the same state as before?

I would greatly appreciate help :)


texts:
I came across the following function in Tensorflow's tutorial on Machine Translation:
BUFFER_SIZE = 32000
BATCH_SIZE = 64
data_size = 30000
train_dataset = train_dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)

I went through several blogs to understand .shuffle(BUFFER_SIZE), but what puzzles me is the fact that a BUFFER_SIZE > DATA_SIZE results in a perfectly uniform shuffling. Neither do I understand what they mean by 'uniform shuffling', nor do I understand how a BUFFER_SIZE> DATA_SIZE is even possible.
From what I understand, tensorflow keeps a BUFFER_SIZE of elements, selects a random element and adds the next input element into the buffer. This makes sense if the BUFFER_SIZE is <= DATA_SIZE. But, what happens to the buffer in case we have more number of elements than the size of the dataset? Do we not have some NULL elements? How does it result in uniform shuffling?
Could anyone please explain to me with an example of how BUFFER_SIZE > DATA_SIZE results in a uniform shuffling? And, what exactly do we mean by uniform shuffling?


texts: How to correctly resize input images in a CNN?

texts:
I have a training set composed of images having different width and height, I need to resize them to a fix dimension nxn or nxm before passing them as the input of a CNN.

I would like to know which parameters I need to take into account for correctly choose the new scaled dimensions n and m.


texts: XGBoost: predictive, descriptive (or both) model?

texts: I have trained an XGBoost model for prediction. The algorithm is able to calculate variable importances.  I was asked why I have not analyzed these variable importances and I did not because as I understood XGBoost is rather a predictive than a descriptive model. I am not sure what extent theses importances can be used to capture real importance (I understand that it may be dependent on the context). Am I right that XGBoost is rather a predictive model?


texts: How to seperate text lines from txt files in python?

texts:
I am playing with a .txt file in python for EDA and I want to seperate the lines from this fashion:

Corona Virus = Justin Piper

COVID19 = Piper, Justin
Fed Paper Says Companies May Automate More Jobs During Pandemic https://t.co/SUqrMfpNJY

Donate the money
Might put food on plate
And smile on face

to this fashion:

[['Corona Virus = Justin Piper'],['COVID19 = Piper, Justin
Fed Paper Says Companies May Automate More Jobs During Pandemic https://t.co/SUqrMfpNJY'],['Donate the money
Might put food on plate
And smile on face ']]

I tried .split('\t') but it is taking each line as a new row.


texts:
I tried to develop a number of CNN architectures to train on a 1000-point subset of the "cat-dog" Kaggle training set (meaning, by the way, that all 1000 data points were labeled). I used a 700-150-150 train-validate-test split, and used the following code on an Xception architecture:

pre_xception_model = keras.applications.Xception(include_top=False, weights='imagenet',
                                                 input_tensor=None, input_shape=(224,224,3), pooling=None, classes=2)
for layer in pre_xception_model.layers:
    layer.trainable = False
dropout = Dropout(0.5)(pre_xception_model.output)
flatten = Flatten()(dropout)
output = Dense(2, activation='softmax')(flatten)
xception_model = Model(pre_xception_model.input, output)

xception_model.compile(Adam(lr=.0001, decay=1e-6), loss='categorical_crossentropy', metrics=['accuracy'])

aug = ImageDataGenerator(rotation_range=20, zoom_range=0.15,
    width_shift_range=0.2, height_shift_range=0.2, shear_range=0.15,
    horizontal_flip=True, fill_mode="nearest")
batches = 20

xception_model.fit_generator(aug.flow(X_train, y_train, batch_size = batches), steps_per_epoch = len(X_train) // batches,
                          validation_data = (X_valid, y_valid), validation_steps = 4, epochs = 10, verbose = 1)

xception_prob = xception_model.predict(X_test, verbose=1)
xception_predict = xception_prob.argmax(axis=-1)
cm_xception = confusion_matrix(y_test[:,1], xception_predict)
plot_confusion_matrix(cm_xception, cm_plot_labels, title='Confusion Matrix')

print(f'\nAccuracy = {(cm_xception[0,0]+cm_xception[1,1])/150}\n')

This produced , and the .predict() produced . I'm at a loss as to explain the overfitting; can someone here help me as to where I went wrong on this?


texts: R Combine Multiple Rows of DataFrame by creating new columns and union values

texts:
I have a dataframe in R that looks like this

 ID    APPROVAL_STEP   APPROVAL_STATUS   APPROVAL_DATE     APPROVER
1234      STEP_A          APPROVED        23-Jan-2019     John Smith
1234      STEP_B          APPROVED        21-Jan-2019      Jane Doe

I need it to look like this

 ID    STEP_A_STATUS  STEP_A_APPROVAL_DATE  STEP_A_APPROVER  STEP_B_STATUS  STEP_B_APPROVAL_DATE  STEP_B_APPROVER
1234     APPROVED         23-Jan-2019         John Smith       APPROVED        21-Jan-2019            Jane Doe

And of course, with the original dataframe, any of APPROVAL_STATUS, APPROVAL_DATE, or APPROVER can be NA.

What is the most elegant way to do this? I know how to do it by looping through the unique IDs, grabbing each row, creating new columns, etc.; but is there any way to do this in a more elegant way (e.g., using tidyverse)?


texts: TensorFlow - Image Vectorization - Add New Files

texts:
I am using tensorflow for feature extraction from images. Images are divided into 5 categories, each having various designs (which are labelled and trained to the system).
Model is trained to find similar images from the database, when one image is uploaded. For this, I have converted the entire database into 5 vectors (1 for each category).
Everything is working good till here. However, I receive 10000 new images daily which I want to be added to the database for comparison. According to my research, the Vector File that has been created earlier is some kind of chronological file, where, I can't add new image vectors. That is, I will have to re-vectorize the entire Database + 10000 images daily, to revise and receive new Vector File (which is used for comparison)
Question: Is there a way to add new image files (vectors) to this existing vector file which has all the image vectors of the database?


texts: How to evaluate sequence to sequence models?

texts:
I wonder how to evaluate variable long sequence-to-sequence predictions?
Let us say I have the following $Y$ and $\hat{Y}$

$Y = [["1", "2", "2"], ["3", "2", "2"], ["1", "3", "2", "2"]]$

$\hat{Y} = [["1", "3", "2"], ["3", "3"], ["1", "3", "2"]]$

Shall I use a binary comparison where any mismatch counts as zero and any full match as one? Or shall calculate conventional accuracy by character-wise comparison?

My concern here is that on one hand, if this is a numeric predcition then any digit mismatch spoils the whole number so doesn't really matter where is the mismatch; on the other hand it would be nice which digits tend to miscalculated in order to find ways to improve the training sets.

Addition: the task is a numeric OCR so -- in contrast with a machine translation job where minor mistranslations are tolerable -- any digit mismatch could result significant business problem (different invoice sums for example). Moreover I'd like to know which individual digits tend to be misread more often so I need a way to get a statistic this way also.


texts: Why do my Latent Dirichlet Allocation Topics mix words that never co-occurred?

texts:
I have one corpus of documents on diabetes, another on Leonardo da Vinci, and another on animation and computer graphics.  I combined all of these documents into a LDA and got a topic like the one below.  I'm listing the top 30 terms, in descending order.

TOPIC 83: ['produced', 'company', 'john', 'weight', 'motion', 'information', 'painting', 'original', 'training', 'people', 'health', 'life', 'jesus', 'feature', 'body', 'lucretia', 'computer', 'graphics', 'time', 'madonna', 'story', 'florence', 'type', 'animated', 'paul', 'diabetes', 'animation', 'exercise', 'peter', 'film']

Many of these words do not co-occur with each other in any documents.  For instance, 'lucretia' does not co-occur with 'exercise'.  Yet how are these terms put together in the same topic, in the top 30 words? (30 out of 20K words or so)

Perhaps it has to do with my particular implementation?  I'm using the gensim library for Python.  Or is it a flaw of LDA generally?


texts: Trying to create an RNN with one character memory

texts:
I'm trying to create an RNN with a one character memory. Here is my model base on an M-GRU (Minimal Gated Recurrent Unit) :

$$f_t = \sigma (W_f[h_t,x_t]+b_f)$$
$$\tilde{h_t} = \tanh(W_h[h_t,x_t]+b_h)$$
$$h_{t+1} = (1-f_t)*\tilde{h_t} + f_t*h_t$$

$x_t$ is the input and $h_t$ is the hidden state of the RNN. $f$ is the update gate for the state.

Then I have the first and second elements of the vector $h_t$ have a special meaning to be the strength of setting or getting the memory. $s=h_t[0]$, $g=h_t[1]$

This adds the input to the memory with strength $s$ and then softmaxed

$$m_{t+1} = softmax( m_{t} + s x_t )$$

Then the candidate output is (it seems to generalise less well without the tanh but converges nicer):

$$c = tanh(W_c h_{t+1}+b_c) $$

Then the real output is given by adding the memory with strength g:

$$o_{t+1} = softmax( c + g m_{t+1} )$$

Then it passes $h_{t+1}$ and $m_{t+1}$ onto the next time step.

Then I tried it on some data "21(22)52(33)5421(44)32112(99)1232(33)132..."
where basically it is random numbers except inside the brackets.
Then I try give it an starting string "(x" which if it successfully works it should output "x)" even though it never saw the character "x" before.

It almost works but is not really converging well.  Here's a typical output:

"(xx)7456725(11)9756x89994x548(33)383351051)(44)9887))1039(11))5407(77)953201(99)x1x85)53)(99))3846"

Is there a better model you know of that does this better (but simpler than a Neural Turing Machine)?


texts: Keras P/R metrics at different thresholds during training

texts:
I'm training a binary classifier and I'd like to see Precision/Recall metrics at different thresholds.
Tensorflow 2.3 introduced tf.keras.metrics.Precision and tf.keras.metrics.Recall which take a thresholds parameter, where you can specify one or multiple thresholds for which you want the metrics computed. This all works as advertised i.e.
m = tf.keras.metrics.Precision(thresholds=[0.1, 0.2, 0.3, 0.4, 0.5, 0.6])
m.update_state([0, 1, 0, 1], [0.4, 0.5, 0.3, 0.8])
m.result().numpy()

Returns the precision value at each threshold [0.5, 0.5, 0.6666667, 1., 1., 1.], as per the documentation.
However when passed as metrics to Model.compile I get a single metric regardless of how many thresholds I have.

pr_thresholds = list(np.arange(0.05, 0.95, 0.05))
model.compile(
    'adam',
    'binary_crossentropy',
    metrics=[
        keras.metrics.Precision(thresholds=pr_thresholds),
        keras.metrics.Recall(thresholds=pr_thresholds),
    ]
)

I get
Epoch 34/50
395/395 [==============================] - 22s 54ms/step - loss: 0.4314 - precision: 0.7886 - recall: 0.9008 - val_loss: 0.5113 - val_precision: 0.7434 - val_recall: 0.8769

What's happening here ? Does it always use the default threshold value of 0.5 in this case ?
Is there a way I can get it to display the values for multiple thresholds during training ?


texts: Ngram based Langauge Models learned using an Encoder-Decoder Model

texts:
I have been going through a Ngram based Langauge Model learned using an Encoder-Decoder Model for Email smart compose.

The program output only 1 prediction for given input.
I want to know how to get multiple predictions out of the same.

Here is the link to the notebook: https://nbviewer.jupyter.org/github/PrithivirajDamodaran/NLP-Experiments/blob/master/Gmail_style_smart_compose_with_char_ngram_based_language_model.ipynb

Here, for input sequence : "hi there" the predicted sequence is: ", how are you today?"
But it maybe possible that I have multiple sentences starting with "hi there" in my training dataset. So how do I get all of those?


texts: Validation loss is less than training loss by 5 units. How this result is interpreted?


texts:
What are the  characteristics of the three correlation coefficients and what are the comparisons of each of them/assumptions?
Can somebody kindly take me through the concepts?


texts: building a model that calculate effect of an event

texts:
Say I have a bike sharing company and want to know how events affect number of rides per day.
events vary on:

number of coupons sent per user
expiry date
amount for each coupon.

I have created a df that looks something like this:
deploy  duration  avg_temp  rain  register_count   ...   rent_count
   33     131113      23      2.2       12                   33
   523    931143      25      0         322                  756
   63     231153      26      0         111                  124
   123    566363      22      1.2       334                  345

where rent_count is our target variable.
If I want to add columns that describe events(num_coupons, duration, etc...) how should I design columns?
So at the end I want to build a model that would look something like this:
Y = b0 + b1x1^n + .... + bnxn^n.

therefore I can calculate how much event variable is contributing to Y.


texts: SSD based on ResNet-101 doesn't improve over SSD-VGGNet

texts:
I am training a SSD model for detecting mobile cranes.
The training dataset contains 1000 images and test set over 400 images.
About 200 epochs gave mAP 83%, but my target is 90%.
So I trained SSD-ResNet-101 and it gave less accuracy.

I assume that it is because ResNet-101 is too deep for the size of my dataset.
I consider using ResNet-50 and Inception.
But I don't have time to experiment all the models with different parameter settings.

Is there anyone who has experience in this direction?
Any advice is welcome.

Thanks in advance.


texts:
My questions seems very trivial, but I can't quite grasp it. I am also aware this post asks for opinions and knowhow, but do not know were else to ask. I do have quite a lot of experience solving even somewhat difficult machine learning problems, but never faced a situation were the target is still a parameter in an other function.
I have a function like:
t = ax + by + cz + bias.
And a database with t, x, y, z, m, n, o, p, etc. I expect the optimal a, b, c etc. to be functions of x, y, z, m, n, o, p. But since I do not know a,b,c, etc and only label/target t instead most machine learning methods from SKlearn do not work out of the box.
I though about first running a linear regression to find a, b, c, etc, that that loses all the information differentiating different datapoints. I guess a self coded SGD would be able to solve my issue, but do not know were to start. I also thought about just dropping my final function completly, but I want to conserve some of the information stored within the function.
For example I do know, that a is dependant on different parameters than b and can give a list to each. My thought is that I reduce possible overfitting by giving as many constraints as possible, espacially since my dataset is not that big.


texts:
This question is about detection of a number having multiple digits in a single image.
I have a trained model in tensorflow using Deep CNN for image recognition. the training was done on cropped images which only had a digit and its label.
Now, the task is to identify any number from the image using this trained model.
I would like this model to work on arbitrary sized number. so how do i extract certain features from the image to be predicted so that individual digit can be detected.
For reference this is the link to the google research paper though it does not elaborate on the issue
here is the data
this question is similar but it does not answer fully as digits nay be too small and close to each other.


texts: Effecient way to decompose multiple time series in a data frame and compare the fit of additive and multiplicative models?

texts:
I have a data frame in R that contains time series data of 7 variables that were taken on several hundred different individuals. I want to know if it would be more appropriate to use an additive model or a multiplicative model for each variable.

To give an example, the data is structured something like this:

set.seed(123)
ID = factor(letters[seq(15)])
Time = c(1000,1200,1234,980,1300,1020,1180,1908,1303,
        1045,1373,1111,1097,1167,1423)
df <- data.frame(ID = rep(ID, Time), Time = sequence(Time))
df[paste0('Var', c(1:7))] <- rnorm(sum(Time))

What is an effective way to decompose the data for each variable/ID combination, fit each with an additive model and a multiplicative model, and compare the fits?


texts: I am new machine learning practitioner. I have run fuzzy c-means algorithm on a multi-label dataset (PPI dataset) on the network using skfuzzy python library. I want to evaluate the performance of the fuzzy c-means algorithm on the dataset using overlapped NMI, Omega Index. I found a github repository OvpNMI, but it requires two clsfiles, ground truth labels file and predicted labels file. I created a truth labels file from dataset but unable to get the predicted labels file from the c-means library. What I am able to get from cmean_predict is fuzzy c-partitioned matrix. I have searched the internet but failed to get answers. Can anyone please help me?


texts: How to build a symptom checker and medical diagnose chat bot

texts:
I want to build a chatbot which can diagnose an illness depending upon the symptoms which are given to it. I want to show you an example scenario of how it should work;

Application - A   ,   User - U

  A) Hello, What is your symptom?

  U) Headache

  A) How long do you have this symptom? ( System will list possible
  choices to the user)

  U) 3 Months

  A) Do you also have photo-affinity? ( System will provide simple "Yes-No-I don't" know choices)

  U) Yes

  .

  .

  A) According to your symptoms, you have "migraine".

There are several applications which work in that manner like "Ada Health", "YourMD" and so on.

I want you to show me a way, give me a kick starter where to start. Are there any resources helpful about this topic?

Can you describe briefly, how one would build such a system in terms of machine learning and data science?

After my questions, I want you to show you my way of thinking about how this system can be built. Please give your comments about it as well

Let's say we have a database and there are symptoms - disease wives. Here is the example;

Once the user enters a symptom to the system, my algorithm is going
to find all of the matched diseases from the database, e.g., "headache" is the entered symptom by the user and the matched diseases with this initial symptom are "migraine", "flu", "cold", "tumor", etc...
After that, all of the symptoms which are connected with these new
diseases are added to the question queue, e.g., for "flu" match, new questions about "flu" disease will be asked to the user like "Do you have also sneeze?"
While the user continues to answer questions, the positive symptoms which user has will be recorded. At the end of the question-answer phase, all of the symptoms which are recorded are going to be used in our machine learning model as the input features.

Does that way of thinking hit the right notes? Thanks in advance for all of the answers.


texts: Learning word embeddings by first learning character embeddings

texts:
I was going through various papers for NLU applications(Natural Language Understanding).
There I have observed a common pattern that for a word embeddings, following 3 combinations are used (may be using concatenation or any other technique).

static word embedding (glove, word2vec)
word embeddings learnt from character embeddings
contextual word embedding

Currently I am implementing one of those papers and I want to learn a word embedding from character embeddings (I don't want to use pre-trained char-embeddings like FastText)
learn character embeddings ---> learn word embeddings from char embedding --> train my custom task
So I am thinking what is the best way to do this?
Let's say my data looks like this:

row-1: What a sunny day. Beautiful! I feel happy.

Our task is, we want find embeddings for each word which derived from char embedding.
approach-1]  use char level CNN (1ConvD)

Here features are nothing but unique characters in our data.

I am assuming result of this step will be, I will have an embedding for each
character.

Then I will go word-by-word, e.g. 'sunny' --> 's'+'u'+'n'+'n'+'y' that is take
a sum/max/avg of char embeddings to get a word embeddings.
However this last step doesn't sound correct wrt backtracking operation. It feels disconnected from model chain.
What am I missing here?

approach-2] Use Bi-Dir LSTM
Something like this as described in this Blog:

But few things are not clear from this:

"What a sunny day"
Here how would I know word boundary? In above picture there was just 1 word so concatenation was easy.

Since we are using hidden state as char embedding that means for same letter (e.g. 'C'), embedding will be different in 2 different words. Is my understanding correct?


texts: I have a classic User-Item dataset where each row (i.e., (user, item)) indicates the action of a user clicking/selecting an item. Now, the dataset only provides positive samples and does not specifically indicate whether a user has disliked an item. In order to create a balanced dataset, I would like to create random negative samples (for instance randomly pick a set of items which the user has never clicked). Of course, I can achieve this by writing a program; however, I found this library in python Scikits called imbalanced-learn, which seems to provide various sampling techniques. Can someone provide a small code-snippet that can use this library (or some other python library) to achieve this?.


texts: Which machine learning technique for product ranking/scoring

texts:
I am trying to identify a ML technique to score products based on the number of times the product was "viewed", "clicked" and knowing the "cost per click" for each product. Given the product ID and category ID, how can I proceed to score each product?

I am sure I have to coarse classify them (some have no clicks, but views, some have both, some have none)? Since there are 1000s of products... Any tip?

I guess the technique is also used in e-commerce to design recommender systems, like based on popularity of a product. Any one can shed some light?

*Edit: Though the suggestions here are interesting, still I couldn't figure out best way to do this.


texts: Vertical concatenation in a df based on column value_python

texts:
Is there any way to concatenate vertically a specific column from my df (concat1), considering/filtering values from another column (col_value)?
My df looks like this:
col_value         concat1
data1             x;y;z
data1             d;f;h
data1             p;c;j
data2             s;k;a
data3             a;w;q
data2             o;i;s
data3             e;q;j
data4             d;f;n
data4             q;f;k

Expected output:
col_value         vertical_concat

data1             x;y;z;d;f;h;p;c;j
data1             x;y;z;d;f;h;p;c;j
data1             x;y;z;d;f;h;p;c;j
data2             s;k;a;o;i;s
data3             a;w;q;e;q;j
data2             s;k;a;o;i;s
data3             a;w;q;e;q;j
data4             d;f;n;q;f;k
data4             d;f;n;q;f;k

Many thanks in advance


texts: How to optimize the lambdas of a hybrid loss in a deep learning model

texts:
I am using a generative adversarial deep learning model (GAN) with a hybrid loss represented by a linear combination of four losses with three $\lambda$'s, something like:

$total\_loss = loss_1 + \lambda_1\times loss_2 + \lambda_2\times loss_3 + \lambda_3\times loss_4$.

Is there a way to optimize these $\lambda$'s towards the best performance? provided that the computational complexity is not trivial. If my objective is for $loss_4$ to be minimum, should I use a high value for $\lambda_3$ compared to the other ones? But, how would that affect the overall performance, as all the models used in the GAN might play an important role in the final result.

NB. $loss_1$ is based on MSE measure, while the other losses are based on $l_1$ norm.


texts: What are and how can I unserstand/process geo levels id from this dataset?

texts:
I am using the dataset from this driven data competition: https://www.drivendata.org/competitions/57/nepal-earthquake/data/

There are three features (geo_level_id_1, geo_level_id_2, geo_level_id_3) that indicates the physical land where the observation is, but I don´t know how can I used it for building a predictive model. Even if this kind of data is universal, I mean, if it is a common way to represent places (such as latitude and longitude).

This data and the others features are described here: https://www.drivendata.org/competitions/57/nepal-earthquake/page/136/#features_list

Do you recommend me some reading or study to understand better this issue?

Thanks a lot!


texts: Apply multivariable linear regression to a dataset in pandas with sklearn

texts:
I'm trying to predict the population for states and the country in 2050. My current dataset has values for each state from 1951,1961...2011 in the same table. Here is a sample view:

Row States/Union Territories         1951   1961    1971    1981    1991    2001    2011
0   Andaman and Nicobar Islands   31    64       115     189     281     356    381
1   Andhra Pradesh               31115  35983   43503   53551   66508   76210   84581
2   Arunachal Pradesh             307   337     468     632     865     1098    1384
.
.
.

So i ran a simple regression for values in 2011 and the model works very well. My question here is, how do i run it for an entirely new column(2050), which has no data to compare with in order to test for accuracy or any other metrics? One thing i intuitively tried was to add a new column  as 2050 and put all the values as 0, but then even the predicted values were 0 so that's of no help. I'm new to this so thanks for any help!

Associated code:

data = pd.read_csv("final_doc.csv", encoding = "latin-1")
data.head()
data['2050'] = 0
X = data[['1951', '1961', '1971', '1981', '1991', '2001']].values
Y = data[['2011']].values
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.2)
regressor = LinearRegression()
regressor.fit(X_train, Y_train)
y_pred = regressor.predict(X_test)
fin = pd.DataFrame(y_pred, columns = ['2011'])
fin


texts: Simple implementation of Apriori algorithm in R

texts:
I am preparing a lecture on data mining algorithms in R and I want to demonstrate the famous Apriori algorithm in it.

My question
Could anybody point me to a simple implementation of this algorithm in R?

(I am not looking for a package, e.g. arules, but for comprehensible source code of an implementation from scratch. It also does not help to look at the source code of arules: It just calls a C implementation of the algorithm!)


texts: Tensorflow take ages for tf.cond and eval() - python code (sorry but i asked on Stackoverflow but none answer me)

texts:
I got a problem with TensorFlow and need your help.
My need is calculating tensordot between a vector: 1x512 named face in my code and a faces data: N x 512 named input_faces_data. The code will return the max_value_index if the max_value >= 0.1.
I printed out time stamp to timing each step of function i use:
tf.tensordot()
tf.math.argmax()
tf.cond()
tf.Session()
.eval() -> return last value

My questions:

Why tf.tensordot() and tf.math.argmax() take just 1ms or 5ms with any length of faces data array (3.000 or 1.000.0000 - my examples) but time cost a lot with .eval and tf.cond()?
Why the duration of tf.cond() and .eval() is more longer with longer face data array?

I'm using TensorFlow 1.13.1 and my GPU is GTX 2080 (11GiB).
My Python code:
sess = tf.Session()
with tf.device(Config.GPU.GPU_DEVICE):
    start = time.time()
    dot_array = tf.tensordot(input_faces_data, face, axes=1)
    print("Data length {}".format(len(faces_data)))
    print("Compatition time {}".format(time.time()-start))
    start_max_index = time.time()
    max_index = tf.math.argmax(dot_array)
    print("get max_value_index time {}".format(time.time()-start_max_index))
    start_condition_time = time.time()
    new_max_index = tf.cond(dot_array[max_index] > tf.constant(0.1),
                            lambda: max_index,lambda: tf.constant(-1,dtype=tf.int64))
    print("tf.cond time {}".format(time.time()-start_condition_time))
    temp_max_index = -1
    start_seesion = time.time()
    with sess:
        print("Session time {}".format(time.time() - start_seesion))
        start_eval_time = time.time()
        temp_max_index = new_max_index.eval()
        print("Eval time {}".format(time.time() - start_eval_time))
    print("Total time {}, max_index {}".format(time.time()-start,temp_max_index))

And my outputs:


texts: Cleaning error text strings inside Categorical Variable Columns

texts:
I have a DataFrame (see below - the real dataset is about 10,000 rows, 26 columns) with errors like '?' or '&#-81;' where there should be a string corresponding to a value like a Premise or a Color.
columns = ['Numbers','Premises','Colors','Year']
mat = np.array([
    [10, 'Q', 'Grey', 1942],
    [50, '?', '&#81', 2091],
    [10, 'P', 'Blue', 1814],
    [15, '?', '&#73', 1719],
    [20, 'Q', 'Grey', 2025]
])
df = pd.DataFrame(mat)
df.columns = columns

 #   Column    Non-Null Count  Dtype
---  ------    --------------  -----
 0   Numbers   5 non-null      object
 1   Premises  5 non-null      object
 2   Colors    5 non-null      object
 3   Year      5 non-null      object
dtypes: object(4)

Those error strings are not missing data because they are text and the field is not empty so df.isnull() :
df.isnull()
    Numbers Premises Colors Year
0   False   False    False  False
1   False   False    False  False
2   False   False    False  False
3   False   False    False  False
4   False   False    False  False

How could I filter or detection in a object column of 10,000 rows for unknown string errors?


texts: Multiclass XGBoost train with num classes = 2

texts:
I have a tagged csv file with 5 calsses.
I accidentally trained am XGBOOST model with this input but forgot to change the num_classes to 5, but instead it was still 2.

The model I received seems to be a very good one, I have tested it on several data sets.
I tried to correct the error and set it to 5, that produced a much worse model.

Does anyone know what is the behavior of the training in such a case?
Because I cant really explain why my model is working that good.

Edit:
we are using Confusion Metrix to determine how good is the model:

y_pred = randomized_mse.predict(X_test)

print(confusion_matrix(y_test, y_pred))

we set num_class": [2],
when in fact we have 5 classes in the data set


texts:
I’m new to time series forecasting and I’m trying to implement regression models using both ARIMA and LSTM for a multidimensional and multivariate time series.
The samples are indexed by time, longitude and latitude (independent variables)  and there are two features to predict.
I found plenty of material on how to handle the multivariate aspect of model, but I’m struggling to find anything describing how to address the fact that the index  is multidimensional.
Few articles refer to the use of Dynamic Time Warping, but I’m not clear on how it would apply.
Any pointers would be very appreciated.
Thank you in advance.
Alex


texts: Classification of non-stationary acoustic signals

texts:
I'm currently working on a task of classification of short non-stationary audio signals with length of 1024 elements and sampled at 120 kHz.

I was wondering if there are any special techniques or models for classification of such signals. As far as I know, most of solution for this kind of tasks relies on computation of FFT/MFCC or  spectrograms. But due to  non-stationary nature of features, I can't compute statisticaly reliable frequency based features.

Also approach with CNN as described in the article Raw Waveform-based Audio Classification Using Sample-level CNN Architectures shows promising results, but still doesn't work good enough.

Can anyone advice me another approaches or thoughts where to start from? Will appreciate any help!


texts: What is the best machine learning model to predict a continuous variable where the predictors include categorical, numerical variables and a text?

texts:
The variables include categorical variables like (contains video, author) and numerical variables like (average word length) and a text (combination of words).
I am confused about this because from what I understand only regression can be used to predict continuous variable. However, I was thinking of forming a bag of words array for the text and title and I do have other categorical variables so i am not sure which model would be ideal. Which ML model would be the best to predict a continuous variable?


texts: options available for visualizing a matrix type data frame in R

texts:
I have a data frame looking like the following:
p1-p5 are different products

A-C: price bands
1-3: size bands
e.g. A1: small store selling the expensive that product
     D3: big store selling cheap that product
values are the numbers of stores for each category A1-C2 per product

type    p1  p2  p3  p4  p5
  A1    158 48  18  6   62
  A2    343 151 58  49  55
  B1    400 181 200 29  863
  B2    398 093 448 215 795
  C1    809 217 154 15  422
  C2    877 549 340 46  576

What would it be the best way to visualize that information and which packages are available for that. I have found gridplot but unfortunately I havent managed to try it yet as I have issues with the Rstudio version.

Examples are more than welcome!

I've got that so far from a function found online


texts: When using an unsupervised alogirthm, what is the "learning" part since it belong to machine learning field?

texts:
I had a brief experience with machine learning by using a clustering algorithm, i also read the basic ideas and calculations of a simple classification algorithm.
Now, i would read more about "machine learning" and I found many similar definitions like the following:

"Machine learning is the science of getting computers to act without being explicitly programmed..."
"Machine learning is a type of artificial intelligence (AI) that allows software applications to become more accurate in predicting outcomes without being explicitly programmed... "

My understanding is that the ability to learn and adapting deduction (output) without reprogrammation is the main idea, and based on my personal understanding, this "adaptation" is only possible with "supervised algorithm" with a new training which can permit a change/adaptation/improvement on the output model with the same program and source code.
So based on my understanding again, this "adaptation" and "learning" definition doesn't fit unsupervised machine learning algorithms, since the model with all calculations is fixed and implemented! Any change will need an update on source code!
Therefore, I would have corrections to my misinterpretation, and more clarifications to have a better understanding of "machine learning" and unsupervised/supervised learning relation.


texts: Linear regression NN in Theano using List

texts:
I am quite new to Theano, while having used Neural Networks in MATLAB before.

I have succeeded with reading all my data from my excel files, and create the input and output lists.

To separate each group, both in input and output, I have created a list that contains lists :

input = [[1 , 2 , 2 ], [5 , 5 ,2], [7 ,8 ,8], [7 ,1 ,1]]

The same goes for output.

So my Input[i] correlates the output[i].

If I create an array, would I lose this correlation? Should I convert my input output to array first and definitely?

Additionally, could a matrix be used instead of array in input and target definition in Theano?

This is also the code I used :
list-1 is input and list-2 , output
When I tried to convert my data directly from their format I got this  error :

  Wrong number of dimensions: expected 1, got 2 with shape (29, 9).

import theano
import numpy
import theano.tensor as T
import matplotlib.pyplot as plt
rng = numpy.random

#Training Data
X = numpy.asarray(list-1)
Y = numpy.asarray(list-2)

m_value = rng.randn()
c_value = rng.randn()
m = theano.shared(m_value,name ='m')
c = theano.shared(c_value,name ='c')

x = T.vector('x')
y = T.vector('y')
num_samples = X.shape[0]

prediction = T.dot(x,m)+c
cost = T.sum(T.pow(prediction-y,2))/(2*num_samples)

gradm = T.grad(cost,m)
gradc = T.grad(cost,c)

learning_rate = 0.01
training_steps = 10000

train = theano.function([x,y],cost,updates = [(m,m-learning_rate*gradm),(c,c-learning_rate*gradc)])
test = theano.function([x],prediction)

for i in range(training_steps):
    costM = train(X,Y)
    print(costM)

print("Slope :")
print(m.get_value())
print("Intercept :")
print(c.get_value())

a = linspace(0,10,10)
b = test(a)
plt.plot(X,Y,'ro')
plt.plot(a,b)

Just as an update I used another code here and I got this error which is more or less the same but the expectance is the contrary :

  Wrong number of dimensions: expected 2, got 1 with shape (6,).