texts: Why TF-IDF is working with Sentiment Analysis?

texts:
Word2vec looks excellent to me as representation of corpus for sentiment analysis. It has relations between words etc. TF-IDF has only weight of the word how important it is. Results with sentiment analysis using both of these representation are quite similar ~90%

Why TF-IDF has such a good results?


texts: matplotlib subplots_adjust - meaning of parameters

texts:
What are the meaning of values in subplots_adjust ?

  left  = 0.125  # the left side of the subplots of the figure

The documentation has number 0.125, etc but there is no explanation.


texts: How to load all images using image_dataset_from_directory function?

texts:
I am working on a multi-label classification problem I faced memory issues so I want to use Keras image_dataset_from_directory method to load all images as batch. how to apply multi-label technique on this method.
I have these folders.
['Tomato_BacterialSpot', 'Tomato_EarlyBlight', 'Tomato_Healthy', 'Tomato_LateBlight']
I am generating class names using the below code.
Here the sample code tutorial for multi-label but they didn't use image_dataset_from_directory  technique.
https://www.pyimagesearch.com/2018/05/07/multi-label-classification-with-keras/
 label = imagePath.split(os.path.sep)[-2].split("_")

and I got the below result but I don't know how to use image_dataset_from_directory method to apply the multi-label?

BacterialSpot
EarlyBlight
Healthy
LateBlight
Tomato


texts: How to get vectors of memory cell & the last output of $LSTM$ in keras?

texts:
In this research paper the following paragraph appears,

  The state of every LSTM model is stored in two fixed-size
  vectors of real numbers called the memory cells and the last
  output. Since our LSTM model is trained to predict user’s
  behavior, elements of these vectors are the natural candidates for the user-dependent features. They can be extended
  by the resulting predictions (answers to the questions). This
  way 218 new features are obtained from the memory cells
  (100) and the last output (100) of the second LSTM layer
  and from the final output (18).

I am aware of getting weights of each layer. But how to get these two vectors?


texts: Is the graphic of deep residual networks wrong?

texts:
I am currently wondering if the following graphic of deep residual networks is wrong:

I would say the graphic describes

$$\varphi \left (W_2 \varphi(W_1 x) + x \right ) \qquad \text{ with } \varphi = ReLU$$

The $\mathcal{F}(x)$ does not make sense to me. Assuming both weight layers are simple MLPs without bias, where the first one has a weight matrix $W_1$ and the second one has a weight matrix $W_2$, what is $\mathcal{F}$?

In the text, they define

$$\mathcal{F}(x) := \mathcal{H}(x) - x$$

where $\mathcal{H}(x)$ is "the desired underlying mapping" (whatever that exactly means).

Also, equation (1) seems strange to me:
$$y = \mathcal{F}(x, \{W_i\}) + x$$
In figure 5 they have two weight layers and call this a building block. Why is there only one weight matrix in this equation?

My thoughts

I think the authors could mean
$$\mathcal{F}_i = \varphi(W_i x)$$
In that case, in the image where $\mathcal{F}(x)$ is it should be
$$\mathcal{F}_1(x) = \varphi(W_1 x)$$
and where $\mathcal{F}(x) + x$ is should be
$$\mathcal{F}_2(\mathcal{F}_1(x)) + x = \varphi \left (W_2 \varphi(W_1 x) + x \right )$$


texts:
I have implemented a Linear regression model on a dataset of 7 independent variable and 1 target  with the below 2 approaches

1) Without normalization of the data, resulted in a Mean squared error of 36530921.0123 and R2  value 0.7477

2) With normalization of the data, resulted in a Mean squared error of 5.468490570335696e-10 and R2 value 0.92750882996584. But in this case even the Target variable is also normalized.

Which is the better approach adding new features to the 1st case or considering the 2nd case and it has a better R2 and MSE value.

If we are considering the 2nd case, what is the steps need to be taken to denormalize the target variable.

Thank you.


texts: Why does BatchNorm on FC layers increase my error?

texts:
I am training a deep CNN for multivariate regression, with three fully connected layers on top of the convolutions. I am using Sigmoid activation for FC layers. When adding BatchNormalization (BN) I observed an alidation error increase, see the picture.

Orange: With BN on all FC layers

Blue: Without BN

It seems like without batch normalization the network needs very long to adjust, probably because the different layers are scaled differently and  maybe also the gradient is very small. But I cannot figure out why at the end the network withous BN achieves a lower validation & testing error.


texts: How to discretize a numerival value with predefined ranges in Weka?

texts:
I have imported a csv file into Weka. One of the features has a value with minimum 0 and maximum 160. Now, I want to discretise that value into three ranges as you can see below:

Less than 6 > L

More than 6 and less than 20 > M

More than 20 > H

How can i do that?


texts: How to approach coarse-grained semantic segmentation?

texts:
I’m looking at doing something like semantic segmentation of images but where I only have pretty coarse-grained labels - roughly, for each 32x32 patch, I know if the answer should be “yes”, “no” or “unknown”.
I’d like to start with a pretrained net that is reasonably well suited for this, and then modify it.
This is a bit like FCN-ResNet101 in the torch hub, but I don’t need any of the stuff that converts the feature maps back up to full resolution - and in my case it would probably both be slow and hurt performance since my labels just aren’t that exact.
I’m not sure what the ideal size of the receptive field for each position of the feature map should be, but I suspect it should be a bit on the small side, around 128x128 to 256x256, based on what I know about the data. Receptive fields for resnet/resnext-50 and 101 before the top pooling layer are really really huge by comparison, 483 or 1027 square (most of which comes from padding with typical 224x224 size inputs). In my case I’d like to feed inputs 1024x1024 or larger and have receptive fields not larger than 256x256 with a 32x32 stride - ideally :)
Is there any example in the literature or a well-known dataset of coarse segmentation similar to this? What would be the state of the art?
What is the best backbone network to use as a starting point? Can I use a pretrained classifier network (I rather like the various swsl_resnext models in timm) and just remove the top pooling layer? Is any specific classifier best for this? (resnext vs efficientnet vs...) Or, should I use a network already intended for segmentation? (in that case I’m a lot less clear on what to remove...)
If I use a network with a large receptive field, how should I shrink it? Should I just remove blocks from the top until the field shrinks enough?


texts: Which are the worse machine learning models for text classifications?

texts:
I was looking at text classification, and for curiosity I was searching online for which were the best models for text classifications. About this, I found that they are linear support vector machines and naive bayes.
But which are the worse models to use in text classification? And, if possible, why?


texts:
Assume the existence of a collection of physical parameters and a collection of output variables which may depend on the physical parameters. An example in the training dataset consists of a vector with the actual measurement of the physical parameters, and another vector with the output values which were measured.

To my understanding, I can train a neural network such as a MLP to learn the mapping from the measured input parameters to the measured output values. If a model could be learned and assuming the learned relations are due to causality and not some unfortunate correlation, then for an arbitrary input vector the model could predict the vector of output values. Note that
the relationship between input and output is not a 1:1 map, rather n:1. Different parameter choices may thus lead to the same output.

I want to go one step further and define an interval for each output value. I then want to know how I must set the ideal input parameter values to ensure that the output values are in the corresponding intervals, i.e. how to alter inputs for the output values to be within the ranges. I tried feeding the output values as input and trying to predict the physical parameters, but I have problems due to the ratio input/output. I have 130 physical parameters and only 10 output variables. How can I achieve this with a neural network?


texts: How to protect data from internal data scientists?

texts:
In our company we want to protect data privacy internally. Meaning, we want to find a way to anonymize the data so the data science team members cannot expose it and yet still can use it for modelling.

I googled and read about Pseudonymization. But I mean, is it destroying the data? I didn't find any reliable source using it practically.


texts: predict future sale from past 3 month data

texts:
I have the 3-month sale(branch wise) and I want to predict next month sale. There are huge differences between total sale in between months. The simple average of the month's to predict next month sale is not valuable. delta(change sale in the month) between months is important and take into account.
Any help, tips, pointers, recommendations, or directions would be extremely helpful and I would be eternally grateful!


texts: Smaller alternatives to sklearn that doesn't require scipy?

texts:
I am packaging my model for deployment in aws lambda which has a size limit of 250mb for all dependencies.
Sklearn, if you include its dependencies of numpy and scipy is a huge package.
Are there any alternatives to sklearn that don't require scipy that are smaller than sklearn?
Thanks!


texts: Creating more than one worker nodes for local windows machine

texts:
I am using windows laptop. And I installed apache spark for my laptop. And I try to measure spark performance by changing spark components. because of that I want to create more than one worker nodes for my local windows machine. first of all I want know is possible to add more worker nodes in my windows machine??
Next thing is if it is possible what are the steps I want to follow??
If some one know about this things please tell me because i search lot time with google but I couldn't find any document regarding that(windows machine)??


texts:
I am doing some analysis on document similarity and was also interested in word similarity. I know that doc2vec inherits from word2vec and by default trains using word vectors which we can access.
My question is:
Should we expect these word vectors and by association any of the methods such as most_similar to be 'better' than word2vec or are they essentially going to be the same? If in the future I only wanted word similarity should I just default to word2vec?


texts:
For a RL task that I am trying to solve, for which I train once per day, I have the rewards stored for each of those days, so that I can see the progress on daily basis.
In the beginning of the learning process, the reward for a given state fluctuates quite much. After about 10 days or so, the rewards start to normalize and the fluctuations are very small. In these cases, when I notice very small change in rewards from day-to-day, I can say that I have achieved a kind of confidence for this state and I want to measure this with a number.
Lets say for S1 I have the following data:
Day1 Day2 Day3 Day4 Day5 Day6 Day7 Day8
0.1 0.3 0.4 0.45 0.5 0.51 0.5 0.51

in this case, after day 6 I start having confidence in my results and on Day8 the confidence should be maximal so far. But for example in Day3, the confidence should be low since the fluctuation is quite high.

Do you have any suggestion on how I could measure this?
Thank you!


texts:
Principal components analysis need standardization or normalization?
After some google, I get confused. pca need the scalar be same. So which should I use.
Which technique needs to do before PCA?
Does pca need standardization?  standardized values will always be zero, and the standard deviation will always be one.
Does pca need normalization? range zero to one
or both ?


texts: Question regarding TimeDistributed(conv1D) with kernel_size > 1

texts:
I am trying to understand how conv1D works in keras.
If I try,

timestep = 10
input_dim = 1
newmodel.add(Conv1D(filters=64, kernel_size=1,activation='relu',input_shape=(timestep, input_dim)))
newmodel.summary()

I understand that 1D conv works as shown in the image.

I expect an output layer (None,8,64).

_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv1d_56 (Conv1D)           (None, 9, 64)             192
=================================================================
Total params: 192
Trainable params: 192
Non-trainable params: 0
_________________________________________________________________

Now if I use

   timestep = 10
   input_dim = 1

   newmodel.add(TimeDistributed(Conv1D(filters=64, kernel_size=2,activation='relu'), input_shape=(None, timestep, input_dim)))
   newmodel.summary()

I get the output shape,

Model: "sequential_24"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
time_distributed_35 (TimeDis (None, None, 9, 64)       192
=================================================================
Total params: 192
Trainable params: 192
Non-trainable params: 0
_________________________________________________________________

Here a conv1D layer is applied to each time step of the input, but each timestep of the input is a (1,1) vector.

Since the convolution window is along the time, how can I have kernel_size > 1 because the kernel_size has to be less than the length of the vector which in our case is 1.

How can I make sense of this?


texts: Better way to graph percentage of One Categorical Variable by another in R

texts:
Looking through the Brfss 2013 dataset (a study that recorded many elements of an observers health habits and outcomes), I wanted to see if there was a correlation between smoking often and being an excessive drinker, by seeing the percentage of the three types of Smokers 'Every Day' 'Sometimes' 'None smokers' that drink often.

This is the code I ended up using:
('drnk3ge5' and 'smokday2' are the respective variables for drinking and smoking)

brfss <- brfss2013 %>%
filter(!is.na(drnk3ge5),!is.na(smokday2))

brfss_1 <- mutate(brfss, bngdrnkr = ifelse(drnk3ge5 >= 4,'Yes','No'))
test1 <- brfss_1 %>%
  filter(bngdrnkr == "Yes") %>%
  group_by(smokday2,bngdrnkr) %>%
  summarise(yes_count = n())
test2 <- brfss_1 %>%
  group_by(smokday2) %>%
  summarise(total = n())
test1$total <- test2$total
test1 %>%
  group_by(smokday2) %>%
  ggplot(aes(x=smokday2,y=(yes_count/total))) +geom_bar(stat="identity")

Is there anyway to code this clearer and more easily understandable?


texts:
I have read few articles some say there is no need to have no. of units in output layer = no. of classes
why some say they both should be equal.
My questions are

If no. of neurons = no. of classes. How are the classes mapped to each neuron/unit in the output layer. To elaborate how does Neural Network decide which neuron/unit deals with which class.

After training a neural network using tensorflow on a multiclass classification problem using softmax in out put layer and no. of units in output layer = no. of classes. When I use this trained network on test sample, the output is a numpy array of probabilities of each class. How do I whther the first element of that array represents which class.

If no. of units is not equal to no. of classes can some one share a link of such an example for multiclass problems.


texts: Implementing Dropout for Recurrent Layers in Keras + Theano

texts:
I am looking to implement recurrent dropout (where recurrent connections between memory units of a recurrent layer such as LSTM/GRU/RNN are randomly set to 0) in Keras 2.3.1 on Theano backend on Python 3.6.

As of Keras 2.1.3, dropout for recurrent layers are no longer supported for the Theano backend:

  Documenting this change further: the motivation for removing this feature in Theano is simply that although it would be technically possible to make it work, it would be hacky, i.e. it would reduce code readability, code maintainability, and importantly, it would be bug-prone. Since Theano development has been discontinued, we expect increasingly less Keras users to rely on the Theano backend, and thus the trade-off between supporting RNN dropout in Theano and having a nice and bug-free RNN codebase is tipping towards the latter.

Discussion

Unfortunately, I am unable to use a TensorFlow backend due to some server limitations nor roll back to an earlier version of Keras.

Any suggestions on how this may be implemented?


texts: Bert Fine Tuning with additional features

texts:
I want to use Bert for an nlp task. But I also have additional features that I would like to include.

From what I have seen, with fine tuning, one only changes the labels and retrains the classification layer.

Is there a way to used pre-trained Bert models and include additional features?


texts:
I'm working on an implementation of LSTM neural network to forecast energy consumption. I have a dataset with load, series of weather parameters and indicator of it's bank holiday or not.

I first did a network with input of 24 lag (using function from this tutorial).
So I have a dataset like this, but with 18 variables and from ($t_{-24}$)

   var1(t-1)  var2(t-1)  var1(t)  var2(t)
1        0.0       50.0        1       51
2        1.0       51.0        2       52
3        2.0       52.0        3       53
4        3.0       53.0        4       54
5        4.0       54.0        5       55
6        5.0       55.0        6       56
7        6.0       56.0        7       57
8        7.0       57.0        8       58
9        8.0       58.0        9       59

And I have my LSTM network:

model = Sequential()
model.add(LSTM(50, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(1))
model.compile(loss='mae', optimizer='Adam')

And currently I use var1(t-1) and var2(t-1) to predict var2(t) value.
But var2 is correlated with value of var1. I have value var1(t) and I want to use it to predict var2(t), but I don't know how to do that.

I tried to fit my neural network with var1(t-1), var2(t-1) and var1(t) but it don't match with then input matrix which is (value, lagged hours, number of variables)


texts:
I have a little confusion.

What follows is from Introduction to Statistical Learning (2013) by Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani.

My understanding of what is going on is the following. The black curve is a function, let us say $y=f(x)$.

We have a random variable $g$ which I write as $g=f+\varepsilon$, $f$ plus noise.

The data points are a subset of the plane, $n$ trials of $g$, the (possibly multi)-set $T:=\{(x_i,g(x_i)):i=1,\dots,n\}$.

I imagine (incorrectly as I know that splines are not polynomials) that the yellow curve is a degree one polynomial (flexibility two) that is the best LS fit to the the trials (the training), the blue is the best degree five polynomial (flexibility six), and the green is the best degree $(20+\alpha)$ polynomial (flexibility $20+\alpha+1$).

In my head, the training should be the data points $T$, while the test should be $f$ (as in $f$ is the expectation of the test data).

I understand that the grey line is telling me that increasing the flexibility (degree of the polynomials in my head), allows me to approximate better the set $T$. However, if I have duplicates of $x$ in $T$, say $x_j=x_k=x^*$, with different $g$ values (e.g. something like $(20,3),\,(20,5)$ both in $T$), then I cannot have a polynomial (or indeed spline or any function) $p$ that has $p(x_j)$ and $p(x_k)$ different: $p(x_j)=p(x_k)=p(x^*)$, single-valued.

Therefore, if I have such duplications in the $x$ variable, I cannot reduce the MSE to zero.

In turn, the red line shows, that when we overfit the data with too much flexibility, the fitted curve is (my usage) biased to the training and so will not model well $f$, and so we have this increasing.

The problem I can't square (excuse the pun), is the dashed line.

It says minimum possible test MSE over all models. Whether 'test' refers to $f$ or $T$ this does not make sense to me.

If 'test' here means $f$, well surely this is zero? We can approximate $f$ arbitrarily well with a polynomial of large enough degree.

If 'test' here means the data $T$, we must conclude that $T$ contains $x$ duplicates: otherwise we could fit a polynomial of degree $n+1$ through all the test points and get this to zero. Therefore there must be duplicates, and so, perhaps, this theoretically best fit goes through all the points which are not duplicated, and goes through the average $g(x_i)$ of the duplicated points... and the answer turns out to be one... but then the grey line should not go below this...

Therefore I conclude that the dashed line is the best possible fit to $f$... but why isn't this zero?

Questions:

  Am I right to be confused by this? Is the black $f$ the test or the
  training?
  Am I misunderstanding something else? Perhaps these (smoothing) splines cannot
  well-approximate as well as polynomials?


texts: How can I change the transparency of a histogram plot in Seaborn using Pairgrid?

texts:
I'm using the Kaggle Titanic dataset. One feature is "Embarked", the city the passenger embarked from. The survival rate appears to correlate with it, but I'm worried it may just be correlated with the ticket Fare (which the survival rate definitely correlates with).

I want to plot a histogram of the fares. That would be easy. However, I also want to, on the same plot, have the histograms for the three embarked values (Q,C,S), labeled by different colors.

I've searched but can't figure out how. I can achieve something relatively similar with FacetGrid:

g = sns.FacetGrid(train_df, col='Embarked')
g.map(plt.hist, 'Fare', bins=20)

which gives three separate histograms, almost what I want:

But that's not quite what I want. For example, it's not great for comparing them. I could calculate the averages, but ideally I'd see them overlaid, so the three histograms are on the same plot, and you can tell them apart because they're labeled different colors.

It seems like PairGrid is almost what I want, but it appears to be giving some strange axis for "Fare" (which is not between 0 and 1), and the legend is blank, and the colors are solid (so you can kind of only see the top layer).

g = sns.PairGrid(train_df[["Embarked","Fare"]], hue="Embarked",size=8)
g = g.map_diag(plt.hist)
g = g.add_legend()

I'm very confused and not sure how to even go about finding the right answer. It seems like Seaborn can do anything, some really cool things, but it's just impossible to find the specific thing I want to do for a given circumstance.

How could I achieve this?


texts: What is the complexity of a recurrent neural network?

texts: In particular, how much memory does a recurrent NN require as a function of the dataset size,number of nodes, etc., and how expensive is it to evaluate at runtime given a new test point?


texts:
I'm currently trying to build and train a model for CIFAR data using keras.
My labels should be one-hot encoded.

data.y_train.shape

is (45000, 10).
My model is defined like this:

model = keras.models.Sequential()
model.add(keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(32, 32, 3)))
model.add(keras.layers.MaxPooling2D((2, 2)))
model.add(keras.layers.Conv2D(64, (3, 3), activation='relu'))
model.add(keras.layers.MaxPooling2D((2, 2)))
model.add(keras.layers.Conv2D(64, (3, 3), activation='relu'))
model.add(keras.layers.Flatten())
model.add(keras.layers.Dense(64, activation='relu'))
model.add(keras.layers.Dense(10, activation='softmax'))

Yet, when I try to train the model, I get this error:

ValueError: Error when checking target: expected dense_10 to have shape (1,) but got array with shape (10,)

Does someone know, why the dimensions do not fit?


texts: Creating features from raw accelerometer data

texts:
I have a dataset containing raw 3-axis accelerometer data collected from a users lower leg and I want to create a classification model (as simple as possible) that detects if the user is sat down or standing. I have around 50,000 events (with more coming in) and these events are labeled with the correct posture and have the raw acceleration for the duration of the posture. I've created a handful of features from the raw data (e.g signal mean, range, frequency content etc) for each event but none of these clearly distinguish the events from each other (when visualising the dataset).
Is there a way of automatically generating useful features from labeled raw data that enables good seperation between outcomes?
If this is not possible, is it best to create a feature set with all the features you can come up with and then try to find the ones that explain the most variance between the outcomes? How do you do this? I've looked at PCA and LDA but they don't appear to 'pick' the best features, they just combine them to new components.
Finally, does anyone have any ideas on features that could explain the differences in standing and sitting lower leg movement. I assume that the lower leg can move in more extreme angles when sitting compared to standing but how do you describe this in features.


texts: Attention Mechanism: Why use context vector instead of attention weights?

texts:
In attention, the context vector ($c$) is derived from the sum of the attention weights ($\alpha$) multiplied by the encoder hidden states ($h$), where the weights are obtained by multiplying the decoder hidden state and the encoder states.

$c_i = \Sigma_j^{T_x} \alpha_{ij} h_j$

My question is, why calculate this context vector and simply not forward the attention weights, as these could indicate how much to focus on each of the encoder states.

Could somebody explain the intuition behind this?


texts: Take several points for each period in a machine learning model

texts:
Problem presentation
I am working on a prediction model where I must find out if a boat will go back to the same offshore workplace after spending time in a port. When a boat is in a port, it can stay here for a few hours or a few days and go back to its working location, or it can go to another place to work on another project. So I want to answer the question 'Will the boat go back to the same workplace after leaving the port ?'
To answer this, I have a dataframe with ashore periods, with some features that can help to deduce what the boat will do :
Out[1]:
     boat  ashore_period_id      port start_ashore  end_ashore  workplace_dist  same_workplace_after_ashore
0  boat 1                 1  Le Havre   2021-01-01  2021-01-05             450                        False
1  boat 2                 2   Dunkirk   2021-01-01  2021-01-02              20                         True

Each row corresponds to an ashore period, we can guess that if the boat is in a port very far from its workplace (workplace_dist column), there is few chances that it go back to its workplace. We can also guess that if the boat stay in the port for a very long time, it will probably go to another workplace after leaving the port.
The predictions for the variable 'same_workplace_after_ashore' will be made at several moments of the ashore period, they won't be made only at the beginning or the end of the period. That's why I have decided to take several points in each period in my training set, 1 point per day. This is how the dataframe looks after the transformation :
Out[2]:
     boat  ashore_period_id      port start_ashore        date  nb_day_ashore  workplace_dist  same_workplace_after_ashore
0  boat 1                 1  Le Havre   2021-01-01  2021-01-01              0             450                        False
1  boat 1                 1  Le Havre   2021-01-01  2021-01-02              1             450                        False
2  boat 1                 1  Le Havre   2021-01-01  2021-01-03              2             450                        False
3  boat 1                 1  Le Havre   2021-01-01  2021-01-04              3             450                        False
4  boat 1                 1  Le Havre   2021-01-01  2021-01-05              4             450                        False
5  boat 2                 2   Dunkirk   2021-01-01  2021-01-01              0              20                         True
6  boat 2                 2   Dunkirk   2021-01-01  2021-01-02              1              20                         True

I now have 5 rows for the first ashore period, and 2 for the second, and only the column 'nb_day_ashore' is changing for a same period. I used this second dataframe to train a random forest.
I have pretty good results, but I wonder if this method is correct, I don't know if it is okay to take several points for each period. We can see in the dataframe that we have more rows for the first ashore period than for the second period. It could create a bias in favour of the long period ashore because the longer periods will have more rows, so it will have more weight in the model training and testing.
Question
Do you know it is is okay to do like this ? Or should I process my data with another method ?
Ideas

Instead of taking 1 row per day and per ashore period, I could take only 1 (or several) date(s) that I would choose randomly in the periods. But it would reduce the size of my training set by a lot.
I could attribute weight to ashore periods depending on their durations.


texts: How to use one hot encoding of string categorical features in keras?

texts:
I am dealing with a binary classification problem. The output column of my dataset is already encoded in 0/1. The problem is that I have many categorical features (columns), which are strings and I would like to one-hot encode them.

I have 18 features (few features are integers and others are strings, the categorical ones) and 1 output column.

I tried this:

dataframe = pd.read_csv('basic_df_export.csv', sep=';', encoding = 'ISO-8859-1', header=None)

dataset = dataframe.values
# split into input (X) and output (Y) variables
X = dataset[:,0:17]
Y = dataset[:,17]

# define example
encoded = to_categorical(X)
print(encoded)

but it doesn't work, throwing me this error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-41-318da09d7033> in <module>()
      9
     10 # define example
---> 11 encoded = to_categorical(X)
     12 print(encoded)

~/anaconda/lib/python3.6/site-packages/keras/utils/np_utils.py in to_categorical(y, num_classes)
     21         is placed last.
     22     """
---> 23     y = np.array(y, dtype='int')
     24     input_shape = y.shape
     25     if input_shape and input_shape[-1] == 1 and len(input_shape) > 1:

ValueError: invalid literal for int() with base 10: 'photo'


texts: How to retrieve the clustering results of rpart

texts:
I am using rpart package in order to create a segmentation of my data using decision tree. As final result I want to obtain a classification of my data. For exemple, if the rpart devide data into 3 classes, I want to divide my data onto this three subsets such that I know that row n°1 is in subset 1, row n°2 in subset 3, etc. I can't find how could I get this information from the results of rpart object?

tree.res <- rpart(x ~ ., data, method="class", parms=list(split = "gini"))

I know that I can retrieve the results of this classification as follows:

plot(tree)    #plot the tree
text(tree)
printcp(tree) #Displays CP table for Fitted Rpart Object
predict(tree) #displays prediction results of x variable

How can I get the information on the segmentation of my data set? As first attempt I used :

predict(tree, type = "vector")

Is this a correct approach?


texts: How to model manufacturing shift data with irregular production times?

texts:
Problem Setting:

Let's say there are three shifts a day in a manufacturing plant. The plant suffers from irregular power supply thereby affecting in how it is functioning in time and the shift's production. Therefore, you have the manufacturing process going on for a while and then pauses due to power outage(or lunch break) to be resumed when the power is back on. This on/off pattern of power is pretty much inconsistent and unpredictable.

Objective:

By mining data of the past history, I want to come up with a model that given a point in a day of a shift I want it to be able to forecast the production for the rest of the shift(possibly accounting for the likelihood of experiencing power outages).

I was hoping for some perspective and idea in how to go about making this problem a machine learning problem and some advise in picking techniques amenable to the problem.


texts: What’s the most suitable programming language for AI development?

texts: For the past couple of years I’ve been learning how to use Python to script.  But I would like to start getting into scripting more things like computers and AI.  So, with that said, and please no hate, what in your opinion would be best to script things like that?  For example - JavaScript or c#/c+/c++


texts:
I'm running some experiments with NNs (actually I'm running an LSTM classifier), and I stumbled across a question I haven't found the answer so far.

What do NNs learn first? When we train a network for classifying objects, for example, the accuracy usually starts low on the training set and increases throughout the epochs. My question is, in these early epochs, which examples of my training set the network is correctly classifying? Are they the "easiest" examples or simply the examples that come first? It seems to me that they are the easiest examples, i.e., those that are easily distinguishable from each other.

I'd be happy if there's a mathematical explanation for that. Any help is appreciated!


texts: How to operate dataset from paperswithcode.com?

texts:
I'm trying to use IG-3.5B-17k dataset from paperswithcode.com, but I can't figure out how to do that exactly.
How to do it? I guess I need to use this site API? Am I even able to use this dataset?


texts: Dataset : Handling rows where some column value is not applicable

texts:
I am working on some LTE Data say of shape (100000, 10) .

Now suppose I have a column Jitter Avg. Now Jitter occurs only for any connected call and for call which could not be setup due to some reason Jitter is not applicable.

So for those rows representing failed called setup there is no value for Jitter i.e NaN. I cannot replace it with 0 since that means good quality. All in all any value would not be applicable here.

So how to deal with such situation.


texts:
I have a Convolutional Neural Network, that is structured as binary classifier. I have two relatively standard convolutional/relu/pooling layers followed by 2 layer fully connected network outputting to a softmax with loss layer for the binary classification. However I observed something unusual:

In Version 1 of the network I had both convolutional layers derive 10 features, upon initialisation I had a Cross Entropy Error of around 28.

In Version 2 I upped the number of features of the convolutional layers to 64 features. Despite still having the same fully connected layers and the same softmax, my Cross Entropy Error jumped to 340.0

My question is, why would this happen. Surely the randomness is the same and the softmax with loss function should normalise so that both outputs add up to 1. So why would Cross Entropy suddenly jump so high

My understanding of the cross entropy effects for outputting large numbers was helped by this answer


texts: yolo v4 vs yolo v4-tiny which is better for real time object detection or is that any that is better for real time object detection?

texts: When I am comparing yolo v4 and yolo v4 tiny I notice that the tiny one somewhat significantly done worse it maybe cause by my small amount of data I use for testing so I want to ask in a normal amount of data like coco dataset amount which is better for real time object detection yolo v4 or yolo v4 tiny and is there better opensource alternative that is as accurate or more accurate than yolo v4 but at the same time as fast as yolo v4 tiny? Thanks


texts: How to keep only significant weights in an ANN

texts: My weights are store in a two dimensional matrix. Row i refers to node i in preceding layer and columns in that row are the neurons node i is connected to. I only want to keep some nodes. How do I pick 3 max weights and store it in a separate array while keeping track of which neuron it belonged to. Moreover, is it tested in theory that some weights contribute more than the others?


texts:
Does anyone know of any publicly available GAN implementations to sort numbers with? As in, the input to the generator is an unordered sequence of numbers, and the goal of the generator is to output that sequence in its correct (ascending) order.
This paper claims to have done it, but sadly I do not think they have released their code.


texts:
I'm new to ML and trying to train a SSD300, with some Keras-Code github.com/pierluigiferrari/ssd_keras I found on github.
For training I'm using an own (very small) dataset of objects that are not in any of the bigger known datasets.
My dataset has the following characteristics:

objects have very different sizes in images (from around 20x40 to 250x200)
there is only one class labeld in the images
images are in RGB
all images are sized to fit in 300x300
dataset contains 319 images for training and validation

Now my problem is, that the loss-function for validation doesn't converge, but training loss does. See this image showing the loss functions over the epochs. I trained 120 epochs with 1000 steps each:

When I try to use the trained weights, coming out of this training, I get zero detections in image. It seems like the model didn't learn anything.
I'm using pretrained weights for the underlaying VGG-16 network provided in the github-repository. It is trained on imagenet dataset.
My parameters are as follows:

img_height = 300 # Height of the model input images
img_width = 300 # Width of the model input images
img_channels = 3 # Number of color channels of the model input images
mean_color = [123, 117, 104] # The per-channel mean of the images in the dataset. Do not change this value if you're using any of the pre-trained weights.
swap_channels = [2, 1, 0] # The color channel order in the original SSD is BGR, so we'll have the model reverse the color channel order of the input images.
n_classes = 1 # Number of positive classes, e.g. 20 for Pascal VOC, 80 for MS COCO
scales_pascal = [0.1, 0.2, 0.37, 0.54, 0.71, 0.88, 1.05] # The anchor box scaling factors used in the original SSD300 for the Pascal VOC datasets
scales_coco = [0.07, 0.15, 0.33, 0.51, 0.69, 0.87, 1.05] # The anchor box scaling factors used in the original SSD300 for the MS COCO datasets
scales = scales_pascal
aspect_ratios = [[1.0, 2.0, 0.5],
                 [1.0, 2.0, 0.5, 3.0, 1.0/3.0],
                 [1.0, 2.0, 0.5, 3.0, 1.0/3.0],
                 [1.0, 2.0, 0.5, 3.0, 1.0/3.0],
                 [1.0, 2.0, 0.5],
                 [1.0, 2.0, 0.5]] # The anchor box aspect ratios used in the original SSD300; the order matters
two_boxes_for_ar1 = True
steps = [8, 16, 32, 64, 100, 300] # The space between two adjacent anchor box center points for each predictor layer.
offsets = [0.5, 0.5, 0.5, 0.5, 0.5, 0.5] # The offsets of the first anchor box center points from the top and left borders of the image as a fraction of the step size for each predictor layer.
clip_boxes = False # Whether or not to clip the anchor boxes to lie entirely within the image boundaries
variances = [0.1, 0.1, 0.2, 0.2] # The variances by which the encoded target coordinates are divided as in the original implementation
normalize_coords = True

Can someone please help me by pointing out these questions:

How to interpret the loss function? Is it because of the small dataset or because of wrong parameters or something else?
Do I have to train my own classifier (VGG-16) or can I use the pretrained one even when my objects don't appear in the pretrained dataset?
Do I have to train for a longer time? Means for more epochs?

As additional information:
I already trained a faster R-CNN model with the exact same dataset. It worked quiet good and gives me good results.

I would appreciate any help you can provide!


texts: What visualization I should choose for Monte Carlo simulations in timeline events?

texts:
I wasn't sure if I should open this question in Cross Validated or here. But since the question belongs to a bigger project related with Data Science, I chose this one.

I will present a simplified version of my working project, since the original is too complicated and domain specific.

Let's say that we have a timeline of 1 hour (60 minutes). During this period a job starts running and create user notifications in random points. I have written a Monte Carlo simulation to study this process.

My main questions:

How well those notifications are spread on the period of 60 minutes?
Do we have parts of this timeline that don't contains notifications and are the rest clustered in a specific time?
Changing the random functions in the implementation, how the above answers will be affected?

A pseudo code for the Monte Carlo, which mimic the actual code is:

Repeat one million times
    number_of_notifications  = get_random_number_of_notifications ()
    previous_point = 0
    for i in range(0 to number_of_notifications):
        interval = get_random_interval()
        new_point = previous_point + interval
        previous_point = new_point

Note: In the current implementation, two or more notifications can be at the same minute.

First, I thought that creating a histogram of the specific minutes during the simulations would help me to answer the first question. But then, I realize that I could have one simulation with all the events in the first half and another one in the second half and the histogram misleading that it is well spread.

Then I thought that it might be nice to plot also the min, max, average and std of the intervals in each simulation. But then, would be enough to answer them?

What kind of visualizations should I try to give me insights about the notifications in the Monte Carlo simulation?


texts: Regression with LSTM network: use multiple time series as input

texts:
I've spent a few days on this and am starting to think I'm missing the obvious solution as this doesn't seem like a very uncommon problem.
As an example dataset: I have 100 measurements with each a different constant speed as target and a length of 10000 data points. Every measurement has additional 30 features that are temperatures, airflow, humidity, etc. My goal is to predict the speed depending on the features/conditions.
My initial dataset so far has the shape (100, 10000, 30) as in (amount of measurements, length of measurements, amount of features) which I would convert with a sliding window shape. All the solutions I have found so far create one big dataset which (in my understanding) would mean that the independent measurements are treated as one single measurement.
As the hard transitions between the measurements harm my predictions: How can I train the network without creating one big measurement that includes these 'transition mistakes'?
I want to use a larger part for training and evaluation and then test a few of the remaining measurements on their own. Therefore I presume my input_shape has to fit a single measurement (10000, 30) and the output target as speed has (10000, 1). How do I transform my data into the window shape while keeping the measurements separate and insert it into my network for training/testing?
I've tried using the batch_input_shape but couldn't find a solution how to incorporate this with the sliding window and existing input_shape so I'm not sure if this could even solve my problem. (I've also thought about training a model each per measurement and trying to ensemble them but that seems way too complicated for this kind of problem.)


texts: Keras -- Transfer learning -- changing Input tensor shape

texts:
This post seems to indicate that what I want to accomplish is not possible. However, I'm not convinced of this -- given what I've already done, I don't see why what I want to do can not be achieved...

I have two image datasets where one has images of shape (480, 720, 3) while the other has images of shape (540, 960, 3).

I initialized a model using the following code:

input = Input(shape=(480, 720, 3), name='image_input')

initial_model = VGG16(weights='imagenet', include_top=False)

for layer in initial_model.layers:
    layer.trainable = False

x = Flatten()(initial_model(input))
x = Dense(1000, activation='relu')(x)
x = BatchNormalization()(x)
x = Dropout(0.5)(x)
x = Dense(1000, activation='relu')(x)
x = BatchNormalization()(x)
x = Dropout(0.5)(x)
x = Dense(14, activation='linear')(x)

model = Model(inputs=input, outputs=x)
model.compile(loss='mse', optimizer='adam', metrics=['mae'])

Now that I've trained this model on the former dataset, I'd like to pop the input tensor layer off and prepend the model with a new input tensor with a shape that matches the image dimensions of the latter dataset.

model = load_model('path/to/my/trained/model.h5')
old_input = model.pop(0)
new_input = Input(shape=(540, 960, 3), name='image_input')
x = model(new_input)
m = Model(inputs=new_input, outputs=x)
m.save('transfer_model.h5')

which yields this error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 2506, in save
    save_model(self, filepath, overwrite, include_optimizer)
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/models.py", line 106, in save_model
    'config': model.get_config()
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 2322, in get_config
    layer_config = layer.get_config()
  File "/home/aicg2/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 2370, in get_config
    new_node_index = node_conversion_map[node_key]
KeyError: u'image_input_ib-0'

In the post that I linked, maz states that there is a dimension mismatch that prevents changing the input layer of a model -- if this was the case, then how is that I put a (480, 720, 3) input layer in front of the VGG16 model which expects (224, 224, 3) images?

I think a more likely issue is that my former model's output is expecting something different than what I'm giving it based on what fchollet is saying in this post. I'm syntactically confused, but I believe the whole x = Layer()(x) segment is constructing the layer piece by piece from input->output and simply throwing a different input in front is breaking it.

I really have no idea though...

Can somebody please enlighten me how to accomplish what I'm trying to do or, if it's not possible, explain to me why not?


texts:
hope someone kindly put time here,

my approach is like this:
TFIDF -> LDA -> SVM

I am using LDA to extract topics. I want to do topic modelling and use the topics as features to do document classification.

I know that I have to send the feature vector to SVM,
but my question is that how can I make this feature vector to send to svm?
Is featureValue here is the probability assigned to each word?
My question is kind of the step 3 in the below link that I dont know how to do it

svm on lda

but in this link there is no implementation, that is just explanation.

Many thanks for your time:)


texts:
I have a regression model that I want to make prediction based on values that I will get from an end user.
In my dataset, I have one categorical variable region which I one-hot encoded, which generated 53 new columns (54 regions).
Now my data has the shape 1000x72. I then split into training and testing sets and my model is working fine.
But I'm confused about how my model would predict new values. Since I will only be getting one value for region from the end user, my model will one-hot encode a single value, and it will no longer fit the shape it has been trained on, as it will have the shape 1x18. I'm really confused as in how would I fit it into the model this way... Do I just make 53 other columns and put a dummy 0 in each one??
Sorry if this is a trivial question, I'm very beginner to this and any help would be greatly appreciated!!

region_ohe = OneHotEncoder(categories = "auto", handle_unknown = "ignore")
X_encoded = region_ohe.fit_transform(df['region'].values.reshape(-1,1)).toarray()


texts: SHAP Explanations in case of repeated train/test split

texts:
I am building a XGBoost model with Python and trying to explain it using the beautiful shap package. Apart from calculating SHAP values of each feature, I'd like to show graphs such as the two that follow (respectively a summary plot Figure A and a dependece plot Figure B):

This is easy if I have a single model, with a single train/test split. However, this is risky as I might have a selection bias, so I usually evaluate the accuracy of my models with 500 random train/test splits (then calculate average values of Accuracy and Cohen's Kappa).

If I do so, how can I get the above plots, considering average information from all the 500 XGBoost models?
Three strategies come to my mind, but none seems to solve the issue:

1- Train a model on all sample (without split) and calculate SHAP values on that. I would keep calculating accuracy and Kappa on the 500 models with train/test split.

2- Select the model of 500 with the best fit on test set and calculate SHAP values on that.

3- I can calculate SHAP values for each feature, for each of 500 repetitions and then calculate their mean and standard deviation. Then I can produce an histogram out of that.

N.1 does not seem to be the correct approach. N.2 also seems to be biased, and things might change from one model to the other.
N.3 is what I have been doing so far, as it seems to be the most reliable approach. However, it has the drawback that I cannot produce the graphs that I would like to have (Figure A and B).

Any idea to produce meaningful graphs (A and B) considering information from a repeated train/test split approach?


texts: Isolation forest: how to deal with identical values?

texts:
I am trying to develop my own implementation of isolation forest algorithm. However I don't know how to deal with points that have the same value for a given feature. To better understand the problem, consider this example:
in my dataset I have the following data: (1, 2), (3,5), (3,4)

We may suppose that after one iteration, we get the following split:

      root
     /     \
    /       \
   /         \
(1,2)    (3,4), (3,5)

Now, how am I supposed to deal with the right branch?
If (due to randomness) we decide to split according to the second feature, we should not have any problem because they are different (4 and 5). However what about if we decide to split according to the first feature (which is 3 in both cases)? Should I repeat the random feature selection until I can split the remaining data?


texts: Is it valid to use numpy.gradient to find slope of line as well as slope of curve at any point?

texts:
what is the difference between slope of the line and slope of the curve? Is it valid to use numpy.gradient to find the slope of the line and slope of the curve  at any point?
    #slope of line at any point
    tanθ= y2-y1/x2-x1

    #slope of curve at any point
    tanθ  =dy/dx

is it valid to use numpys np.gradient() to get both slopes of curve and line ? or is it meant only to find the slope of line?
Reference slope of line vs curve


texts: A data set to distinguish man from woman?

texts: Is there a set and/or a model that is available freely in the Internet that can tell man from woman based on input image?


texts:
Is there a way to train your machine learning model in the cloud? Or does it really have to be batch training? i.e. (Pull some data on SQL, then feed that to the model)

What i was thinking is implementing my own model from scratch, use Stochastic Gradient Descent to update the parameters for every row from the database.


texts: Let's say if I have 2 TB of data, what is the best sample size to pick? I understand that there is a limit on how much RAM/processing power I have and hence I should make my sampling decision around that. But let's say if the processing power is not a concern for me right now. What would be a good way to approach my sample size?


texts: Word frequencies in unbalanced case-control dataset

texts:
I have a case-control cohort for which I'm doing analysis of clinical notes. The ratio of cases to controls is 1:4. What I'm looking at is the relative frequency of certain words (e.g. overdose, suicide) for each group and from documents spanning a 30-day period.
For days numbered 1 to 30, for cases and controls, I want to count the number of times each word is mentioned in all documents for that day. I wish to visualise the results in a plot of some description.

There are two factors of imbalance in the data set:

the number of individuals in each group, i.e. the case-control ratio
the number of documents for each group - cases have on average more documents per day than controls

The first is easily mitigated by dividing control counts by 4, but what is a reliable way to offset the difference in relative number of documents between cases and controls?


texts: Value Error: Operands could not be broadcast together with shapes - LSTM

texts:
I am trying an LSTM model using tensorflow following this tutorial .

I am having trouble understanding why am I getting an error in my test set when I try to invert scaling for forecast (line 86 in the tutorial).

After loading the dataset and created featured, following is what I did for reframed dataset,

# split into train and test sets
values = reframed.values
split_point = len(reframed)- 168

train = values[ : split_point, :]
test = values[split_point: , :]

# split into input and outputs
train_X, train_y = train[:, :-1], train[:, -1]
test_X, test_y = test[:, :-1], test[:, -1]

# reshape input to be 3D [samples, timesteps, features]
train_X = train_X.reshape((train_X.shape[0], 1, train_X.shape[1]))
test_X = test_X.reshape((test_X.shape[0], 1, test_X.shape[1]))

print(train_X.shape, train_y.shape, test_X.shape, test_y.shape)
>> (2399, 1, 39) (2399,) (168, 1, 39) (168,)

# design network
model = Sequential()
model.add(LSTM(50, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(1))
model.compile(loss='mae', optimizer='adam')

# fit network
history = model.fit(train_X, train_y, epochs=50, batch_size=72, validation_data=(test_X, test_y), verbose=2, shuffle=False)

# make a prediction
yhat = model.predict(test_X)
test_X = test_X.reshape((test_X.shape[0], test_X.shape[2]))

inv_yhat = np.concatenate((yhat, test_X[:, 1:]), axis=1)

After this step, the following code that has scaler.invert_transform throws the error.

inv_yhat = scaler.inverse_transform(inv_yhat)

  ValueError: operands could not be broadcast together with shapes (168,39) (41,) (168,39)

Shape,

test_X.shape
>> (168, 1, 39)

yhat.shape
>>(168,1)

inv_yhat.shape
>> (168,39)

Any help or sugeestions would be appreciated!


texts: Tool for annotation of images for semantic segmentation

texts:
I have been searching around for a software tool, that I can use for annotating images. More specifically I want to do annotation to be used for semantic segmentation, meaning I want to create masks. I want to be able to create training data for applying a segmentation CNN (like for instance U-net).
However I have been digging around the internet, and I have tried out some options. But I have not really found anything that seems to do the trick.
I do not expect to have many classes, but I probably want to be able to be able to annotate more than two classes (something, background). And of course every pixel has to belong to a class.
So what I am looking for is a relatively simple tool that either lets you mark areas of a class with a brush or that lets one mark polygons and then fill those, and then the outside the polygons as background. So can anyone point me to anything like this? And share their experience?


texts: Where can I download historical market capitalization and daily turnover data for stocks?

texts:
There are plenty of sources which provide the historical stock data but they only provide the OHLC fields along with volume and adjusted close. Also a couple of sources I found provide market cap data sets but they're restricted to US stocks. Yahoo Finance provides this data online but there's no option to download it ( or none I am aware of ).

Where can I download this data for stocks belonging to various top stock exchanges across countries by using their ticker name ?
Is there some way to download it via Yahoo Finance or Google Finance ?

I need data for the last decade or so and hence need some script or API which would do this.


texts: Is dimension reduction helpful to select features for a classification problem?

texts:
Let's say I have a data set but I don't know what features are relevant to solve a classification/regression problem.

In this case, is it worth/good to use a dimension reduction algorithm and then apply a classification algorithm? Or can I  just select "randomly" my features by using my common sense and then try to tune my algorithm next?

Also if someone has some explanation of a dimension reduction "in real life with real use case" it would be great because I feel my comprehension of dimension reduction is wrong!


texts: Is 10,000 images for one class of spectrograms good enough for music classification model?

texts: I'm debating on using a DNN or CNN for this classification, but I have 10,000 spectrograms of songs. I was wondering if I need more because the images have a low variants.


texts: Clustering unique visitors by useragent, ip, session_id

texts:
Given website access data in the form session_id, ip, user_agent, and optionally timestamp, following the conditions below, how would you best cluster the sessions into unique visitors?

session_id: is an id given to every new visitor. It does not expire, however if the user doesn't accept cookies/clears cookies/changes browser/changes device, he will not be recognised anymore

IP can be shared between different users (Imagine a free wi-fi cafe, or your ISP reassigning IPs), and they will often have at least 2, home and work.

User_agent is the browser+OS version, allowing to distinguish between devices. For example a user is likely to use both phone and laptop, but is unlikely to use windows+apple laptops. It is unlikely that the same session id has multiple useragents.

Data might look as the fiddle here:
http://sqlfiddle.com/#!2/c4de40/1

Of course, we are talking about assumptions, but it's about getting as close to reality as possible. For example, if we encounter the same ip and useragent in a limited time frame with a different session_id, it would be a fair assumption that it's the same user, with some edge case exceptions.

Edit: Language in which the problem is solved is irellevant, it's mostly about logic and not implementation. Pseudocode is fine.

Edit: due to the slow nature of the fiddle, you can alternatively read/run the mysql:

select session_id, floor(rand()*256*256*256*256) as ip_num , floor(rand()*1000) as user_agent_id
from
    (select 1+a.nr+10*b.nr as session_id, ceil(rand()*3) as nr
    from
        (select 1 as nr union all select 2 union all select 3   union all select 4 union all select 5
        union all select 6 union all select 7 union all select 8 union all select 9 union all select 0)a
    join
        (select 1 as nr union all select 2 union all select 3   union all select 4 union all select 5
        union all select 6 union all select 7 union all select 8 union all select 9 union all select 0)b
        order by 1
    )d
inner join
    (select 1 as nr union all select 2 union all select 3   union all select 4 union all select 5
    union all select 6 union all select 7 union all select 8 union all select 9 )e
    on d.nr>=e.nr


texts: Is it possible to classify documents of corpus using labels?

texts:
I have a corpus of 23000 documents that need to be classified into 5 different categories. I do not have any labeled data available to me, just freeform text documents and labels(yes, one-word labels, not topics).
So I followed a 2-step approach:

Synthetically generate labeled data (using a rule-based labeling approach, obviously the recall is very low, ~ 1/8 documents are labeled)
Somehow, use this labeled data to identify labels for other documents.

I have attempted the following approaches for step 2:

Topic modeling on data classified using rules to extract significant terms and using significant terms to label the remaining documents.
Finding significant terms using sentence embedding
Using sentence embedding as features for my classifier

But I haven't been successful in getting good results for my document classifier. Are there any other methods that can be used to classify the documents?
All help is greatly appreciated.


texts: Can Reinforcement Learning work for Dutch auctions?

texts:
I had the following idea of Reinforcement Learning:

There is the RL agent and the environment. The agent can execute actions, sends them to the environment and receives an observation and a reward (a float).
Accounting problem: The reward might be delayed.
Learning algorithms: Q-learning is one example. It basically tries to learn the mapping between State x Action to reward and works like this

Can Reinforcement Learning work for a setting similar to Dutch auctions?

The agent can make 3 actions: Increase, keep or lower the price
The agents goal is to sell the item for the highest price. So the agent is the auctioneer.
The auction item does not change. The bidders are anonymous and thus don't give a hint about their behavior.
This means the observations are the same, only the time to the end gets lower - independently of the action of the agent.


texts: how to interpret a high AUC value but a low F1 score after upsampling minority class?

texts:
I use to gradient boosting to classify my data between default and paid. The data is very imbalanced where default is in minority. The fisrt classification report from sklearn gradient boosting classifier give:

             precision    recall  f1-score   support

   Defaulted       0.56      0.03      0.05       364
        Paid       0.87      1.00      0.93      2420

   micro avg       0.87      0.87      0.87      2784
   macro avg       0.71      0.51      0.49      2784
weighted avg       0.83      0.87      0.82      2784

and an AUC = 0.614

To addressed the imbalence issue, I use SMOTE to upsample the default class
and I am getting an AUC = 0.941

However, the classification report wost f1-score for the default class:

              precision    recall  f1-score   support

   Defaulted       0.38      0.01      0.02       364
        Paid       0.87      1.00      0.93      2420

   micro avg       0.87      0.87      0.87      2784
   macro avg       0.62      0.50      0.47      2784
weighted avg       0.81      0.87      0.81      2784

Is this possible and how should I interpret or understand this difference between the worst f1-score but a better AUC?


texts: Why is the GAE usually caculated looping backwards over the rewards?

texts:
I've recently implemented a PPO-Agent in Python, which uses the GAE as Advantage estimator. Almost all Tutorials on this matter calculate the Advantage looping backwards over the rewards:
gae = 0
n = len(rewards)
advantage = np.zeros(n, dtype=np.float32)
for i in reversed(range(n)):
    delta = rewards[i] + self.gamma * values[i + 1] * (int(not is_terminal[i])) - values[i]
    gae = delta + self.gamma * self.gae_lambda * (int(not is_terminal[i])) * gae
    advantage[i] = gae

However it is possible to calculate the Advantage without a reversed for loop:
n_rewards = len(rewards)
advantage = np.zeros(n_rewards, dtype=np.float32)

for t in range(n_rewards):
    discount = 1
    a_t = 0
    for k in range(t, n_rewards):
        a_t += discount * (rewards[k] + self.gamma*values[k+1] * (int(not is_terminal[k])) - values[k])
        discount *= self.gamma * self.gae_lambda
    advantage[t] = a_t

So why do most implementations do the reversed for-loop ?
My first thought was that discount *= self.gamma * self.gae_lambda becomes more and more inaccurate for long iterations, due to floating point precision. However, I was unable to confirm that within my code ...


texts: torch.save(the_model, PATH) vs torch.save(the_model.state_dict(), PATH) - model loading incorrectly for one method

texts:
I just now noticed that the model does not get loaded correctly if I use the the_model.state_dict() method to save it. On the other hand, using the_model method, it gets saved exactly the same and I am able to load it back correctly. Why could this be happening?
My loading code:
fcn = FCN()
if torch.cuda.is_available():
#     checkpoint = torch.load("./model/fcn_200.pt")
#     fcn.load_state_dict(checkpoint)
    fcn = torch.load("./model/fcn_200_freez.pt")

else:
#     checkpoint = torch.load("./model/fcn_200.pt", map_location=torch.device('cpu'))
#     fcn.load_state_dict(checkpoint)

    fcn = torch.load("./model/fcn_200_freez.pt", map_location=torch.device('cpu'))

ind = range(0,5)
visualizePrediction(fcn, voc2012.train_images[ind], voc2012.train_labels[ind])
visualizePrediction(fcn, voc2012.val_images[ind], voc2012.val_labels[ind])

O/P with the_model:

This is what I saw on google collab when I had trained the model.
O/P with the_model.state_dict():

It's literally random each time! :/


texts: GridSearchCV with SVM estimator AUC score not reproduced on SVM run

texts:
I've run GridSearchCV to determine 'best parameters' for a linear SVM, and then passed these [in a dictionary along with non-tuned parameters] into a new SVM.

I've printed out the parameter dictionaries and confirmed that the output of the grid search feeds the parameters correctly into the new SVM (sklearn.svm.SVC()), and yet where the grid search achieved AUC of 0.89, on the same dataset I achieve just 0.5, and "no predicted samples".

To confirm, I set the param_grid to a 1x1 grid (i.e. no searching, just running 5-fold cross-val using SVM estimator) and it still "does well" only to drop to 0.5 AUC score afterwards.

Can anyone please help diagnose this disparity?

(Note that this code has a reduced logspace parameter space, to reduce running time after I found the most performant values)

import numpy as np
from sklearn.model_selection import StratifiedShuffleSplit, GridSearchCV

def SVMParamTune(model_class, X, y, params):
    from sklearn.svm import SVC
    C_range = np.logspace(-2, -2, 1)
    gamma_range = np.logspace(3, 3, 1)
    param_grid = dict(gamma=gamma_range, C=C_range)
    # If the configuration contains other parameters, use these
    # These are the 'constants' to parameterising the SVM
    for param_name in params:
        if param_name not in param_grid.keys():
            pass
            # param_grid[param_name] = params[param_name]
    cv = StratifiedShuffleSplit(n_splits=5, test_size=0.2, random_state=42)
    # NB the SVC estimator provides a `score` method,
    # meaning GridSearchCV uses mean accuracy (from this estimator's score)
    grid = GridSearchCV(SVC(), param_grid=param_grid, scoring='roc_auc', cv=cv, n_jobs=10, verbose=10)
    grid.fit(X, y)
    print("The best parameters are %s with an AUC score of %0.2f"
                  % (grid.best_params_, grid.best_score_))
    print(grid.best_params_)
    for param in params:
        if param in grid.best_params_.keys():
            params[param] = grid.best_params_[param]
    print(params)
    return params

This is run, I see a good result, yet the result doesn't make it out of the tuning process for reasons unknown.

m_params = SVMParamTune(m_class, X=data[0], y=data[1], params=m_params)

The class (sklearn.svm.SVC, stored in m_class) is then run with these updated parameters, and yet gives different results

model_run = m_class(**m_params)

Switching StratifiedShuffleSplit to StratifiedKFold to be consistent with the latter SVM run made negligible difference so cannot be the cause.

Update returning the model itself also gives the same changed AUC score - the model itself cannot be reproduced, on the same data...


texts: Image classification with different number of training image for each class

texts:
I'm trying to train neural network for image classification with 4 different classes:

Cars (22k training examples)
Building (8k training examples)
Pedestrian (5k training examples)
Trees (1k training examples)

The problem is that the number of training images is biased toward one/two class(es).

I'm wondering if there is a way to train neural network in terms of the number of training examples per class?

Do I have to limit the number of training examples per class to the minimum number of all classes?


texts: Feature Importance and Partial Dependence plots seem to disagree?

texts:
I need some help understanding my partial dependence plots for binary features passed to a GradientBoostClassifier when comparing them to the feature importances. For some background, my goal here is to investigate user churn. My classifications are: 0 = not churned, 1 = churned.

First of all when I plot the feature importances I get some interesting results which do not agree with the feature importances from my random forest. I put this down to algorithmic differences since many do align or are similar, but my specific concern is the binary feature 'registeredEmail', which is clearly very important in the random forest model, but not in the other (gradient boost, as shown). This is not my main concern, just adding in case relevant to my question.

Secondly, and this is the part I am most confused by. Why does my feature importance chart show that 'playerInAlliance' (another binary feature) is significantly more important than 'registeredEmail', since when I check the partial dependencies there is a much steeper slope for 'registeredEmail'. I would interpret this as the churned prediction being highly influenced by a player not having a registered email, compared to being less likely to predict churn if the player does have their email registered. Is this correct?

Please note these plots are in descending order of feature importance, from left to right. Also ignore the title.

Comparing this to 'playerInAlliance' I can see that no matter if the player is in an alliance or not, this feature is influential in predicting a churned player. I don't understand how to interpret this, as if there's no real difference between players churning when in an alliance compared to having no alliance, then why is my GradientBoostClassifier considering it to be a highly important and highly dependent feature?

Finally, if I plot both of these binary features against each other on one plot, we can see that dependence on a player not having a registered email is massively higher than if the player is in an alliance or not. So why is the registeredEmail feature less significant in terms of importance?

TL;DR: How do I interpret Binary features in partial dependence plots?

Any help appreciated.


texts:
I am trying to implement fuzzy logic system to classifiy dataset of 12 inputs and 1 ouput. I wanna understand as first taks to fuzzify inputs how Can we set intervals or we need to segment inputs first in order to fuzzify them

below is an example of fuzzification but the chose of the intervals is not clear.

Any suggestion or explication will be appreciated

# Generate fuzzy membership functions
qual_lo = fuzz.trimf(x_qual, [0, 0, 5])
qual_md = fuzz.trimf(x_qual, [0, 5, 10])
qual_hi = fuzz.trimf(x_qual, [5, 10, 10])


texts: What are the features to distinguish two short term signals (time series)

texts:
Imagine that I have a set of short term time series data (signals) with different shapes and parameters. I would like to extract/select a set of features (parameters/functions/indeces) to identify each sequence and distinguish between them. Such features are simple features e.g. max and min value, mean, variance, start and end value, etc. or more complex ones e.g. entropy, no. peaks, power spectrum, etc.
I am looking for more "representative" features to express the signals and distinguish them. What features do you know (in signal processing or data mining fields) that can represent a parameter in signals?
For example, in the following image, there are two signals with the same parameters such as mean, min-max values. But what are the features that differ them (for instance no. peaks, but generally what else?)


texts: SPC vs Autoencoders in anomaly detection

texts: Considering the usage of Autoencoders in anomaly detection of time-series data, why SPCs (control charts) have lost their charm? Are there any advantages with Autoencoders and disadvantages with SPCs?


texts: Intuition: why are ReLu activations boundary lines linear?

texts:
What is the reasoning behind ReLu boundary lines appearing linear when plotting in 2D? Does this generalize to higher dimensions in that boundary lines in large dimensions are linear hyperplanes as well?

For instance, in tensorflow's playground any time ReLu is chosen as the activation function, the resulting boundary line is made of lines.

I'm failing to see the connection between the underlying neurons outputting only z or 0 and how that connects to the output plot.

Thanks very much for any thoughts/help!


texts:
I am writing an ETL pipeline for geospatial data of the form
place_name,address,longitude,latitude,id_linking_to_other_dataset
As the last step in the pipeline, I would like to apply manual transformations submitted by reviewers. Some of these transformations might be (borrowing from Google maps suggest edits docs):

Change a place's name, location, or the id linking it to another dataset
Mark a place private or non-existent
Mark a place as moved or duplicated

I don't have a ton of records (about 5000) but would like to manage this manual correction using best-practices. Ideally, corrections could be version controlled and applied as the last step in an ETL pipeline (even if other parts of the pipeline change).
There are lots of good tools handing off data annotation for ML, but I'm not seeing resources for this type of correction. Thoughts on useful tools?


texts:
i want to build a music recommender system. i have user_id,song,play_counts triplets as my data.I want to do it with collaborative filtering in with i will have USERS row and SONGS columns.It is very similar to movie recommendation but instead of rating,i now have play counts.So, cn i use play count as my matrix values...i will be then using linear regression to predict features of both users and songs simultaneously.
Will it work with play counts to predict music?
Please answer ...i am confused!


texts:
I have a dataset which consists of multiple user ratings. Each rating looks similarly to:

| Taste | Flavour | Look | Enjoyed | ..... | Tag    |
|-------|---------|------|---------|-------|--------|
| 4     | 2       | 2    | 3       | ..... | Banana |
| 5     | 4       | 1    | 2       | ..... | Apple  |
| 3     | 1       | 4    | 1       | ..... | Pasta  |
| ....  | ....    | .... | ....    | ....  | ....   |

The columns contain ranks for each row. The task is to clusterize rows,
e.g. I would like to find something similar to:

cluster 1: Banana, Apple
cluster 2: Pasta, Spagetty
....

We use HDBSCAN with edit distance metric to find clusters, and it works more or less.
The problem, however, is that there are too few features (12 in total) to have "good" clusters.
Therefore I would like to somehow account for the information from "Tag"
in clustering. The idea is to calculate embeddings for each tag and use them as features.

What I'm not certain about is how to include these new features? I would
like the clustering to be primarily determined by the original features. The dimension
of embeddings is much larger than the dimension of the original features, and
the metric on these features is different (e.g. cosine similarity). Therefore, I would like to answer 2 questions:

What will be a proper method to combine these heterogeneous features?
How to properly select the weight for the "Tag" feature? Ideally, I would not like to just postulate it


texts: MobileNet and MobileNetV2: Bad Inference Results

texts:
unfortunately I am having subjectively bad results in inference with pre-trained models of both MobileNet v1 and v2:

from keras.applications.mobilenet_v2 import MobileNetV2
ConvNet = MobileNetV2(input_shape = None, include_top = True, weights = 'imagenet', input_tensor = None, pooling = None, classes = 1000)

I have a local copy of these networks for the corresponding image size (224x224), depth multiplier 1.0 and weights trained for ImageNet.

After loading a model of MobileNetV2, I am exectuing a classification on random images from ImageNet or Google Images. Almost always the top-1 classification does not make any sense, for example very often I get the suggestions of a "Shower Curtain" or a "Pillow", although this is obviously not the case.

Testing with other models (VGG16, ResNet50) and changing only the model type (keeping the same parameters), I obtained correct or at least more understandable results that were also consistent among these two different models:

ConvNet = VGG16(input_shape = None, include_top = True, weights = 'imagenet', input_tensor = None, pooling = None, classes = 1000)

Having correct results here with the other models, I assume that my script is working correctly.

My question is: Has anybody ever experienced these issues with inference with MobileNet or MobileNetV2? And/or do you have any idea why this error occurs and how to solve it?

I appreciate any answer, please also consider seemingly trivial solutions since I am still quite a newbie ;)

Thanks a lot,
Tim


texts: Train CNN-RNN network for multi label video classification with sliding window technique

texts:
I’m implementing a model in which a CNN model is used to extract feature sequences from videos , and RNN is used to analyze the generated feature sequences, and output a multi label classification result. Frame-wise annotation of actions is not given but only the temporal order so the model is being trained with CTCLoss.
I want to use sliding window tecnique to generate more video segments (overlapping windows) to capture dependencies over different timesteps.
Should i use x.unfold( ) in the input video or in the generated features after the cnn and before the rnn input ?
does it have any effect in the backward operation ?


texts: Python library to detect a bank/financial institution name in a string

texts: I would like to extract bank names from a given text like wells Fargo, chase....is there a python library for this? I know there is entity tagger in space and flair but they only identify the entity (org/person)


texts: How to apply StandardScaler and OneHotEncoder simultaneously in Spark Machine learning?

texts:
I try to create a machine learning model, linear regression, to predict a price of a diamonds.

All examples that I found online do not have a step with scaling of data, using MinMaxScaler or StandardScaler. But I personally think that this is an important step in ML. The Spark guide shows this feature, but separately from OneHotEncoding. Moreover, because the result of OneHotEncoding in Scala is different from Python (for instance, it looks like this: (4,[3],[1.0]) in Spark, while in Python it is a combination of 1 and 0), I am confused where to apply StandardScaler - after Indexing and before OneHotEncoder, or after OneHotEncoder or in some other step?

My code after I cleaned data a little bit. This code works, but predictions are far from being good. I use Scala dataframe and pipeline:

import org.apache.spark.ml.feature.{VectorAssembler,StringIndexer,VectorIndexer,OneHotEncoder}
import org.apache.spark.ml.linalg.Vectors

val cutIndexer = new StringIndexer().setInputCol("cut").setOutputCol("cutIndex")
val colorIndexer = new StringIndexer().setInputCol("color").setOutputCol("colorIndex")
val clarityIndexer = new StringIndexer().setInputCol("clarity").setOutputCol("clarityIndex")

val cutEncoder = new OneHotEncoder().setInputCol("cutIndex").setOutputCol("cutVec")
val colorEncoder = new OneHotEncoder().setInputCol("colorIndex").setOutputCol("colorVec")
val clarityEncoder = new OneHotEncoder().setInputCol("clarityIndex").setOutputCol("clarityVec")

val assembler = (new VectorAssembler()
                  .setInputCols(Array("carat", "cutVec", "colorVec", "clarityVec", "depth", "table", "x", "y", "z"))
                  .setOutputCol("features") )

val scaler = new StandardScaler().setInputCol("features").setOutputCol("scaledFeatures").setWithStd(true).setWithMean(false)

val Array(training, test) = df_label.randomSplit(Array(0.75, 0.25))

import org.apache.spark.ml.Pipeline

val lr = new LinearRegression()

val pipeline = new Pipeline().setStages(Array(cutIndexer,colorIndexer, clarityIndexer,cutEncoder,colorEncoder,clarityEncoder, assembler, scaler, lr))

val model = pipeline.fit(training)

val results = model.transform(test)

The subset of data:

+-----+-----+---------+-----+-------+-----+-----+----+----+----+
|label|carat|      cut|color|clarity|depth|table|   x|   y|   z|
+-----+-----+---------+-----+-------+-----+-----+----+----+----+
|  326| 0.23|    Ideal|    E|    SI2| 61.5| 55.0|3.95|3.98|2.43|
|  326| 0.21|  Premium|    E|    SI1| 59.8| 61.0|3.89|3.84|2.31|
|  327| 0.23|     Good|    E|    VS1| 56.9| 65.0|4.05|4.07|2.31|

Thanks in advance!


texts:
I'm trying to solve a multilabel classification (dataset is tweet text) using a combination of BERT and CNN. As a benchmark, I'd compare it to other word embeddings, one of which is Word2vec. After numerous tries, it seems that Word2vec-CNN keeps outperforming BERT-CNN by a slight bit, here's a result from my last try:
Word2vec-CNN
precision (macro): 0.89
recall (macro): 0.87
f1-score (macro): 0.88
accuracy (test set): 0.81
hamming loss: 0.062

BERT-CNN
precision (macro): 0.86
recall (macro): 0.88
f1-score (macro): 0.87
accuracy (test set): 0.74
hamming loss: 0.073

Question is:

Could it be possible that Word2vec (or any static word embeddings) outperforms BERT (or any contextual word embeddings)? If so, what is the rationale? If there's any research paper on this it would be really helpful.
If not, what could possibly be the cause?

FWIW: Model is trained using TensorFlow-Keras (I kind of suspect this is SOMEHOW caused by how TF-Keras calculates its metrics but I still haven't figured out why and, if any, a solution), and both embeddings are pretrained (BERT model was trained on a bigger corpus, around 200:1).


texts: The implementation of overlapping clustering (NEO-K-Means)

texts:
I am researching on overlapping clustering (Clusters are non-disjoint).I found that Neo-K-Means is probably the state-of-the-art now.But, when I tried  implementing the algorithm with the multi-label data set (music-emotion/scene).I hadn't got the high result as declared in the paper (My results are around 0.4 F-measure , the paper declare 0.55 for music and 0.626 for scene ). In spite of , I had initialized the experiments with the best situation for K-Means (centroid seeds are means of each class and the total cluster assignments are equal to reality) . I wonder what wrong with my implementation method , orDo I have to do something extra for getting higher result ?

PS. I have found the further research of Neo-Kmeans , but I think I should clear this point before go further.

This is my code

while (count < TIMES) {

            DC = new ArrayList<DistanceCollection>();

            for (int i = 0; i < K; i++) {

                cluster[i] = new Cluster();

            }

            for (int i = 0; i < dataList.size(); i++) {

                for (int j = 0; j < K; j++) {

                    DistanceCollection dist = new DistanceCollection();
                    dist.dataNum = dataList.get(i).dataNum;
                    dist.clusterNum = j;
                    dist.distanceFromCluster = euclidean(centroids[j], dataList.get(i));
                    DC.add(dist);

                }

            }

            // sort the distances for argmin(i,j) checking
            Collections.sort(DC, new DistanceCollectionComparator());

            int totalAssignment = 2585;
            int assignedCluster = -1;
            int assignmentCount = 0;
            int[] dataSelectionCheck = new int[2407];
            int dataMatrix[][] = new int[6][2407];

            for (int i = 0;  assignmentCount < dataList.size(); i++) {

                int clusterNum = DC.get(i).clusterNum;
                int dataNum = DC.get(i).dataNum;

                if (dataMatrix[clusterNum][dataNum] == 0 && dataSelectionCheck[dataNum] == 0) {

                    cluster[clusterNum].dataMembers.add(dataList.get(dataNum));
                    dataMatrix[clusterNum][dataNum] = 1;
                    dataSelectionCheck[dataNum] = 1;

                    assignmentCount++;

                }

            }

            for (int i = 0; assignmentCount < totalAssignment; i++) {

                int clusterNum = DC.get(i).clusterNum;
                int dataNum = DC.get(i).dataNum;

                if (dataMatrix[clusterNum][dataNum] == 0) {

                    cluster[clusterNum].dataMembers.add(dataList.get(dataNum));
                    dataMatrix[clusterNum][dataNum] = 1;

                    assignmentCount++;

                }

            }

            for (int i = 0; i < K; i++) {

                if (cluster[i].dataMembers.size() > 0) {

                    for (int j = 0; j < centroids[i].features.length ; j++) {

                        double accumFeaturesValue = 0;

                        for (int k = 0; k < cluster[i].dataMembers.size(); k++) {

                            accumFeaturesValue += cluster[i].dataMembers.get(k).features[j];

                        }

                        centroids[i].features[j] = accumFeaturesValue / cluster[i].dataMembers.size();

                    }
                }

            }

            count++ ;

        }


texts:
I have trained a few NLP models, measured their performances and now I want to create a final model for production trained with all the data I have available.
I'm working in text classification and I'm using transformers with the library PyTorch Lightning. The issue here is that I can't find an example of how to turn off the validation step (with pytorch lightning) and after several google searches I was not able to find discussions about that.
So, now I'm having second thoughts and questioning if I'm thinking correctly. Is it common practice to train the production model with the full dataset without the validation? Or should I just use the best model I have already trained? If so, why?


texts: Derivative of activation function used in gradient descent algorithms

texts:
Why is it necessary to calculate the derivative of activation functions while updating model( regression or NN) parameters? Why is the constant gradient of linear functions considered as a disadvantage?

As far as I know, when we do stochastic gradient descent using the formula:

$$\text{weight} = \text{weight} + (\text{learning rate}\times (\text{actual output} - \text{predicted output}) * \text{input})$$

then also, the weights get updated fine, so why is calculation of derivative considered so important?


texts: How to get K most different rows in csv?

texts:
We have boring CSV with 10000 rows of ages (float), titles (enum/int), scores (float). How to select 1000 most different rows? I look for a general solution that would work for more than one case.
What do I mean by different:

We have N columns each with int/float values in a table.
You can imagine this as points in ND space
We want to pick K points that would have maximised distance between each other.

So if we have 100 points in a tightly packed cluster and one point in the distance we would get something like this for three points:
 or this

It looks like an ND point cloud "triangulation" with a given resolution yet not for 3d points... So how to select K most distant rows (points) from N (with any complexity)?


texts: How to test/validate unlabeled data in association rules in R?

texts:
I produced association rules by using the arules package (apriori). I'm left with +/- 250 rules. I would like to test/validate the rules that I have, like answering the question: How do I know that these association rules are true? How can I validate them? What are common practice to test it?

I thought about cross validation (with training data and test data) as I read that it's not impossible to use it on unsupervised learning methods..but I'm not sure if it makes sense since I don't use labeled data.

If someone has a clue, even if it's not specifically about association rules (but testing other unsupervised learning methods), that would also be helpful to me.

I uploaded an example of the data that I use here in case it's relevant: https://www.mediafire.com/?4b1zqpkbjf15iuy


texts: How to generate abbrevations from shortend words in medical records

texts:
I have text files which contains medical history of a patients and would want to extract information out of it.

Basically what want is generate english text of abbrevations,semantic category region,location and relation also correct spelling mistakes if any by parsing medical history records.

I have google around for open source tools and found cTAKES & Metamap.However could not find python api for the same.

Can someone suggest how to make use of these tools or interact via python?and also is there a better way to extract information using regular epression..etc.

Sample sentence:
55 yr M comes to the ED with c/o of a chest pain that started 1 hr ago,
The cP are a/w SOB.

Expected o/p
55 yr(year) (M)male comes to the (ED)emergency department with (c/o)complaint of a chest pain that started 1 hr(hour) ago,The (cP)chest pain is associated with (SOB)short of breathing.

Apart from generating abbrevations,also need to cature temparatures,levels such as 5/4,semantic category region,location and relation also correct spelling mistakes in the clinical notes.

While am waiting for UMLS license,just wanted to check can above functionality be achieved through UMLS/cTaked/Metamap? or cutsom script needs to be written in conjunction with python/NLP/RE and open source clinical abbrevations dictionary(please suggest)?


texts: How would one modify CycleGAN in order to map a distribution to itself?

texts:
CycleGAN can map between two different distributions $X$ and $Y$ with cycle consistency. This is done with generator functions $F: X \mapsto Y$ and $G: Y \mapsto X$, such that $||G(F(x)) - x||_1 \approx 0$, where $x \in X$, and $||F(G(y)) - y||_1 \approx 0$, where $y \in Y$.

What if instead, I want to have a function $H: X \mapsto X$, such that:

$||H(H(x)) - x||_1 \approx 0$
But... $||H(x) - x|| > k$, where $k$ is some minimum distance.

So the second point is key here. In plain English, I want to map $x_1 \in X$ to $x_2 \in X$ such that $x_1$ and $x_2$ look like different examples of the data distribution. I'm not so sure the formulation I provided matches this plain English description, but the latter is what I'm really asking.

Also, I'm not sure if cycle consistency is totally needed for what I'm asking, but I think it might be. That's because I want $x_2$ to have the same arrangement of high level objects as $x_1$, but with different finer grain features.

Finally, to give you a more visual representation of what I'm looking for: the following image (taken from the CycleGAN paper) shows oranges translated to apples in the same arrangement.

I'd like to take a dataset of maybe 10 different types of spherical fruits and treat it all as my $X$, and get a function $H$ which can take a picture of an arrangement of say apples, and then produce a similar spatial arrangement of another fruit in the dataset.


texts: How can implement this equation in code(Pytorch)

texts:
I am implementing a NLP paper. I am struck here with this equation, how to convert it to PyTorch?
Sigma(h0*W1*h1 + h1*W2*h2)

Here W1&W2 are weights. H0,h1 and h2 are of shape (batch, hidde_dims).
I tried:
nn.Sigmoid(hm*(nn. Linear()(hm-1) +nn. Linear()(hm+1)))

but dims doesn't match
equation taken from this paper


texts: Data visualization multiple choices multiple condition conditions

texts:
Say I have a element that base on a certain condition (1, 2, 3 or 4), it chooses a decision (A,B or C). What diagrams can be used to present this data if I have more than one thousand elements so that:

One can see what are the most popular selection based on a particular condition.
Easy to identify which element change their decision if the condition is changed.

Any ideas? Any suggestion or hint would be very much appreciated!

UPDATE: I have implement parallel coordinates (suggested by @Oxbowerce), but for instance for condition3, one cannot see how much of elements have chosen the different decisions. Also, for instance from condition1 to condition2, how much have moved from decision "2" to decision "1"? it is not visible from the plot.
More suggestions will be very welcome!


texts:
Are there some resources for filters specifically applicable in big data applications?

Particularly, are there major differences between filter design for other domains and filter design for data mining?


texts: What is cohen kappa metric, implementation in Python?

texts: Can somebody explain indetail explanation on Quadratic Kappa Metric/cohen kappa metric with implementation in Python


texts: Can autoencoders take time series into account?

texts:
Here, I read the following:

  The first key to understanding is that HTM relies on data that streams
  over time (...) By contrast, conventional deep learning uses static
  data and is therefore time invariant.  Even RNN/LSTMs that process
  speech, which is time based, actually do so on static datasets.

On the other hand, on I read the following:

  We can assert that our LSTM Autoencoder is a good weapon to extract
  importan unseen features from time series

What is true now? Are LSTM or autoencoders or combined suitable for time series analysis? Or is HTM better suited for that purpose, indeed?

The main purpose is to detect anomalies emerging through time before they happen.


texts: Server log analysis using machine learning

texts:
I was assigned this task to analyze the server logs of our application which contains exception logs, database logs event logs etc. I am new to machine learning, we use Spark with elastic search and Sparks MLlib(or PredictionIO).An example of the desired result would be to be able to predict based on the exception logs gathered to be able to predict which user is more likely to cause the next exception and at which feature(and bunch of other stuff to keep track and improve optimization of the application).

I have successfully been able to ingest data from ElasticSearch into spark and create DataFrames and map the data needed. What I would like to know is how do I approach the Machine Learning aspect of my implementation. I've been through articles and papers that talk about Data preprocessing, training the data models and creating labels and then generating predictions.

The questions I have are

How do I approach transforming the exiting log data into numerical vectors which can be used to datasets to be trained.
What algorithms do I use to train my dataset(with the limited knowledge i've gathered the past couple days, i was thinking bout implementing linear regression, please suggest which implementation would be best)

Just looking for suggestions on how to approach this problem.

Thank You.


texts:
In a work, I have to benchmark different algorithms to fill in missing values in time series.
I insist on the fact that this is imputation and not forecasting.
In my case, I have access to 15 years of complete temperature data from 20 stations.
I have different algorithms that, knowing the position of missing values, try to complete the missing data.
However, these algorithms have some parameters to be set. So I want to implement a classical method such as k-cross fold validation.
Moreover, these algorithms have to be tested in a classical case of time series completion. Indeed, here is a typical drawing of time series data completion:

Each line is a time series.
(note that some series are and must be complete for the benchmark)
In red are the missing data to be completed, and in green are the unknown data. In practice, these algorithms use the notion of space-time to complete missing data.
However I am facing a problem. Indeed, the method of k-cross fold validation takes randomly data and it is this purely random character which poses me a problem. Indeed, as shown on the drawing, some algorithms could work better if I take my data completely randomly. Whereas in reality, they will only be tested on a template as on the image.
Indeed, I know in advance that some of my algorithms work very well when blocks of several years of known data are present and don't work at all when purely randomly selecting the known data.
Do you have an idea of how to parameterize my algorithms to be optimal on a data framework such as the one on the image?
Thanks in advance
(Feel free to ask me questions if I haven't been clear enough)


texts: Is there any kaggle competition for finding the feature for affecting revenue?

texts:
I have a question about finding the most significant feature of the data that affects the company revenue. I have data set that contains car share company data (car model, completed trips, lead_time, trip length, trip revenue, delivery_fee, net_revenue ...) columns.

Is there any kaggle competition out there doing EDA (Explotary data analysis) not prediction for finding the most significiant feature that affects the net_revenue or sales ? Could not find and uber of lyft data competition for this question!

I would appreciate If you happen to know such data set and share the link with me!

Thanks in advance!