texts: How to write a generator for keras fit_generator?

texts:
This question is a further step of this question.

My data inputs are tens of .csv files, I have already read csv input data until the following format:

# train_x is data, train_y is label
print(train_x.shape)       # (2000000,10,100)  3D array
print(train_y.shape)       # (2000000,)    labels

I already can fit & evaluate them using:

model.fit(train_x, train_y, batch_size=32, epochs=10)
model.evaluate(train_x, train_y)

It works well if the dataset is LESS than RAM size. But if dataset is too BIG then "large dataset do not fit the memory". Most online suggestions are to use fit_generator( ) instead of fit( ) (also suggested from keras website).

fit_generator(generator, steps_per_epoch=None, epochs=1, verbose=1, callbacks=None, validation_data=None, validation_steps=None, validation_freq=1, class_weight=None, max_queue_size=10, workers=1, use_multiprocessing=False, shuffle=True, initial_epoch=0)

How to write a generator function (the 1st parameter of fit_generator)?

I only know the generator function aims to feed data batch by batch.

  As the name suggests, the .fit_generator  function assumes there is an underlying function that is generating the data for it.

What should be included in this geneator function? What should be returned? Any related example?

Mark: I have read several online examples (e.g., this and this). They use images as example, which is not my case (csv data only), and not easy to understand.


texts: what exactly is the hessian matrix in xgboost?

texts:
i would like to understand a bit more the mathematics behind xgboost. I understand that the hessian is the second partial derivative of the loss function, i originally thought this was with respect to the log of the odds. But what i do not understand is how it is a matrix, and how it applies to a matrix of training data where columns are feature and rows and number of samples.
https://www.youtube.com/watch?v=ZVFeW798-2I&t=126s i have watchrd this helpful video but this is looking at just one feature 'dosage'. if i have say many features how do the loss functions change. i am trying to view the bigger picture here but am struggling to see how we build a xgboost tree from matrix.  below shows the hessian calculation.

is the loss function calcaulted per row? per feature? please help on maths of this , greatly appreciated! how is it calcaulted per row, since a row is a combination of features.


texts: Preparing multiple training time-series for Keras LSTM regression model training

texts:
I have training data organised in a numpy array in which:
* column is feature - last one is the target,
* every row is one observation.

The thing is that this 2D array consists of around 15 concatenated series of data of unequal length.I would like to use it for LSTM network training. Is it possible to train LSTM network in Keras using series of unequal length? If so, how am I supposed to transform my 2D training data array into valid 3D array?


texts:
I'm looking for a general approach to solve a problem that is best demonstrated by the following example.

You mix six different batches of orange juice and you know the acidity, the sugar content, the concentration of volatile particles in each batch of the juice, the growing location of the fruits, and other data. You mix those batches, sell the product, and measure how well it was accepted by the market. Tomorrow, you decide to mix four other batches, the next day you will mix two batches, etc: the number of components in your product isn't constant. The goal is to take the information about each component and to predict the outcome metric.

If we could compute the average value of each input parameter, I would start by computing the average value of each parameter in the mix, and trying to use it as the X data in an ML algorithm. However, some of the parameters cannot be added. In our example, such a parameter can be the plantation age, plantation location, cultivar of the fruit, etc

What are the approaches to solve this problem? Does it belong to a certain studies problem class ?


texts:
I am working on an image similarity network. I have around 90,000 pairs of images contain an equal number of positive and negative samples. For learning the similarity between image pairs, I used the pre-trained network (VGGNet-16). After getting feature vectors I merged them and use their merged feature vectors as an input to the similarity network.
The similarity network contains two convolution layers and four fully connected layers. I trained the network and only after first epoch I got the following outcome:

    Epoch 1/1
    1382/1382 [=======] - 1557s 1s/step - loss: 0.3431 - acc: 0.9556 - val_loss: 0.3334 - val_acc: 0.9874

I am a newbie in neural networks. Could someone comment on this result after a single epoch? I saw networks require 100’s of epochs to achieve accuracy up to 95%. Is this overfitting? Looking for suggestions.


texts: Loss given Activation Function and Probability Model

texts:
Can someone please explain this one?

  For output units a good trick is to obtain the output non-linearity
  and the loss by considering the associated negative log-likelihood and
  choosing an appropriate (conditional) output probability model,
  usually in the exponential family. For example, one can typically take
  squared error and linear outputs to correspond to a Gaussian output
  model, cross-entropy and sigmoids to correspond to a binomial output
  model, and − log output[target class] with softmax outputs to
  correspond to multinomial output variables.


texts:
Each layer of a neural network learns features of the input data. The first layer learns low-level features (e.g. edges in images). Each subsequent layer learns more abstract features. Then the features from the last hidden layer are combined for classification or regression.

How to make sure that the features learned by a neural network are not correlated?
I think that I can force a neural network to learn correlated features in the following way. Remove the output layer, so that the last hidden layer produces $n$ outputs. Then train this network on $n$ target variables that are not independent but are linear combinations of say two or three independent targets  (maybe with some noise added to avoid true linear dependence). As neural networks are universal approximators, it should be possible to train such a network. However, in that case, all neurons in the final layer will be redundant except those two or three neurons that correspond to the linearly independent targets.
This redundancy can be present in each layer. How to avoid it? Of course, I can check the correlation of neuron outputs after the training, remove the correlated neurons, and retrain the network but it looks inefficient. How to make the network learn uncorrelated features from the beginning?
What comes to my mind is using different and independent (e.g. orthogonal) activation functions for each neuron within the layer. Are there such networks? Usually, activation functions are taken to be the same.
EDIT: I tried to test if dropout can make learned features less correlated. I did not find that it does (or I am doing it in a wrong way?).
This is a toy problem that I tried: $y = a^2+b^2$ where $a$ and $b$ are uniformly distributed between 0 and 1.
a_train = np.random.uniform(low=0.0, high=1.0, size=10000)
b_train = np.random.uniform(low=0.0, high=1.0, size=10000)
y_train = a_train**2 + b_train**2
X_train = np.hstack([a_train[:,np.newaxis], b_train[:,np.newaxis]])

The input features are not correlated.
np.corrcoef(X_train[:,0], X_train[:,1])[0,1]

0.002136928470247475

Neural network:
model = keras.Sequential(
    [
        layers.Dense(32, activation="tanh", name="layer1", input_dim=2),
        layers.Dense(16, activation="tanh", name="layer2"),
        layers.Dense(2, activation="tanh", name="feature_layer"),
        layers.Dense(1, name="out_layer"),
    ]
)
optimizer = tf.keras.optimizers.RMSprop(learning_rate=0.001)

model.compile(loss='mean_squared_error',
                optimizer=optimizer,
                metrics=['mean_squared_error'])

model.fit(X_train, y_train, epochs=100, batch_size=128, validation_split = 0.2, verbose=1)

feature_extractor = keras.Model(
    inputs=model.inputs,
    outputs=model.get_layer(name="feature_layer").output,
)

features=feature_extractor(X_train).numpy()

The correlation between the outputs of the next-to-the-last layer:
np.corrcoef(features[:,0], features[:,1])[1,0]

-0.7527464552820806

After adding dropout,
model = keras.Sequential(
    [
        layers.Dense(32, activation="tanh", name="layer1", input_dim=2),
        layers.Dropout(0.2),
        layers.Dense(16, activation="tanh", name="layer2"),
        layers.Dropout(0.2),
        layers.Dense(2, activation="tanh", name="feature_layer"),
        layers.Dense(1, name="out_layer"),
    ]
)
model.summary()

The correlation becomes
-0.8645152290698691


texts: Doing a cox regression, how do I analyze a continuous variable in which a lower result is worse?

texts:
I have this variable that is clinically worse the lower it is. Instead of interpreting, for example, a hazard ratio of 0.9 as "per unit increase in variable x, there is a x% reduction in risk for mortality", I want to get a >1 HR (for ex 1.05) and say "per unit decrease in variable x there is a 5% higher risk for mortality".
How do I handle this using SPSS?


texts: Which reinforcement learning methods can be trained off-policy?

texts:
My understanding is that:

Value-Based methods such as DQN, C51, Rainbow DQN can naturally be trained off-policy using a Replay Buffer, without having to account for any kind of off-policy correction.
Deterministic Policy Gradient methods, such as DDPG or TD3 can be trained off-policy, as they don't integrate over actions.
Stochastic Policy Gradient methods can't, on principle, be trained off-policy, as they need to estimate the gradient using the current policy. However using importance sampling, you can correct that off-policiness, eg the way TRPO or ACER does.

So in the end all the major methods: value function approximations, deterministic policy gradient, and stochastic policy gradient, can be trained using a replay buffer.

Is my understanding correct?


texts: Performance of CNN based deep models with number of classes

texts:
How does a given deep cnn model performance vary with number of classes in tasks such as classification, object detection segmentation?

For example mobilenet v2 gives around 90% accuracy on classification task using PASCAL VOC; now if we train the same network with just 2 classes (say human and dog) how will it effect accuracy, speed and model size?(say, using tensorflow).Also what will be the case in tasks like object detection and semantic segmentation(in particular)?

Can we make the network smaller to learn from the smaller dataset(2 classes) to achieve same or better accuarcy & speed?


texts:
I am looking for ideas on how to proceed with a situation. I have historical data of appointments for many users. I can (easily) predict their future behaviour (whether the next appointment will be positive or not)/

I now need to build profiles, i.e., classify a user as "good" or "bad" user. So, I won't be predicting future appointments, but just say "this user is a good/bad user", and then compare with, e.g., a monthly behaviour.

Any ideas where I could look for information on how to proceed?

I apologize for my vague first question. I will try to be more clear.

I have ~200k users, with their doctor appointment history. As demographic, I only have their age and gender. The rest of the variables relate to their appointment: time of the appointment, which doctor (service), day of the appointment, and so on. I also have whether they went or not to the appointment (show / no show).

The assignment consist on classifying the users as "good" or "bad", i.e., whether they will go or not to the appointment. I do not have to predict if they will go to the next appointment, just to have a list of the users classification. By doing this, if user A, whom I classified as "good", calls for an appointment, I know he is "good" and he will make it to the appointment. I do not to take any action on him.

Again, I do not have to predict future appointments, or the behavior of a new user, just to classify the existing ones.

I hope now it is a bit more clear. If not, let me know. Thank you!

I know perhaps the information I am giving is a bit vague, but I didn't know if to go into full detail or not. If you want me to share more info, just let me know.

Thanks!


texts: What Framework To Use for Asynchronous Algorithms?

texts:
I have a problem with an extremely large dataset (who doesn't?) which is stored in chunks such that there is low variance across chunks (i.e., the chunks are sort of representative). I wanted to play around with algorithms to do some classification in an asynchronous fashion but I wanted to code it up myself.

A sample code would look like

start a master
distribute 10 chunks on 10 slaves
while some criterion is not met
 for each s in slave:
  classify the data inexactly using some kind of iterative algorithm and return to master
 master waits for any 2 slaves to report the classifier, averages the classifier and sends it back for the slaves to continue

What framework do I use? Hadoop, Spark, Other?

If I was doing this in pure-C, I would use pthreads and have a very fine control over threads, locks and mutexes. Is there any analogous framework in this distributed data-science environment?


texts: How is the Fully Connected Layer of a cnn network trained?

texts: When training a cnn network, only the filters are updated, but how are the Fully Connected Layer weights updated?


texts: The upper range of a collected dataset is most likely accurate, but the rest may suffer biased omissions: How to call this phenomenon?

texts:
Background: In collecting a dataset of a specific unit ordered by a numeric variable, it is possible that the upper 'cloud' of the dataset is correct, while the 'tail' seems inaccurate.
I can thus trust the upper bounds of the dataset, while I deem the mid- and lower ranges to be rather questionable.
Question: Is there a data scientific term for this phenomenon? If so, how is it called?
Example: I use various sources to gather a list of major publishers of scholarly journals. In total, I found 150 publishers that publish at least 50 journals.
At the top, I found the publisher AAA with 3.000 journals, BBB with 2.500 journals, CCC with 1.900 journals, DDD with 1.500 journals etc.
With my industry knowledge,  I can confirm that the top 20 is likely to be accurate.
However, in the lower ranges, there are publishers like XXX with 51 journals, YYY with 50 journals, ZZZ with 50 journals etc. Many of them are rather obscure, and even as an expert I may have never heard of them. I can imagine that the 'tail' is rather inaccurate with large-scale omissions, such as publishers from the Global South.
I thus tend to trust the top 20 or so of that list, but not the ones that rank between #21 and #150.


texts: How to do k-fold validation with classifiers?

texts:
I want to cross-validate a model that plays the card game below (see image).

I trained the model on a dataset of 1000 games, with the goal to maximise the profit of each game. It works great on the training dataset, but I want to k-fold validate it, to ensure that it generalises well.

How can I apply k-fold validation in this case?. This model is not a regressor. It is a classifier, with a variable reward/penalty moreover. So I cannot calculate the MSE of validation folds, there is no MSE here. How can I apply k-fold validation in this case?


texts: Deep Reinforcement Learning - mean Q as an evaluation metric

texts:
I'm tuning a deep learning model for a learner of Space Invaders game (image below). The state is defined as relative eucledian distance between the player and the enemies + relative distance between the player and 6 closest enemy lasers normalized by the window height (if the player's position is $(x_p,y_p)$ and an enemy's position is $(x_e,y_e)$, the relative euclidian distance is
$\frac{\sqrt{(x_p-x_e)^2+(y_p-y_e)^2}}{HEIGHT}$ and HEIGHT is the window height). Hence the observation space dimension is (10+6), which results in an input of my deep neural network of 16 units.

My agent doesn't seem to learn (reward function doesn't increase) and I thought I'd check the mean Q values, which are the output of my main deep neural network, and, instead of increasing, I've remarked that the mean Q values stabilizes (as in figure below) instead of increasing. I've modified many tuning parameters (batch size, neural net architecture and parameters...) but I still have same problem. Any idea why the mean Q values wouldn't increase ?

Here are some results about the learner:


texts: Discarding correlation among inputs in a neural network

texts:
I am working on a problem with 4 inputs and 1 continuous output variable. The sum of all values of the 4 input variables is always 1.

  a1+a2+a3+a4=1

So, they are correlated.

My question is: should I use all 4 variables for neural network training? Or, should I use any 3 of them to get rid of correlation? Is there any problem if I use all 4?


texts: I have a pandas dataframe and i need to clear all the special characters other than space

texts:
Input:

import pandas as pd
df=pd.read_excel("OCRFinal.xlsx")
df['OCR_Text']=df['OCR_Text'].str.replace(r'\W+'," ")
print(df['OCR_Text'])

Output:

The excel removes all the special characters along with the space. But i dont want space characters to be removed


texts: Why use Gradient Descent when Gradient just solve the problem? (With Neural Nets)

texts:
My knowledge is that gradient actually gets to the global minimum and gradient descent try to take steps to the direction that he judges to be the lowest.
I know that calculate the gradient from complex functions is non trivial problem, but let's say that we are a using neural net with a quadratic cost function, would be that hard to calculate the gradient and get actual global minimum?


texts:
I've a sequence of recurring events I want to group together into representing different operation activities of the underlying process.

1) These events might potentially have an order in their occurrence; or may be not. I want to explore and investigate if any relationship existing between the events as such. Any other better than Hierarchical clustering?

2) I might want to build a model that can determine the operational activity based on the events it recognized as belonging to the clusters representing them operational activities. May be HMM, RNN etc.

I've been researching for an algorithm that lends itself to the problem I'm trying to solve. I came across a few that might be adapted to be able to solve the problem like HMM, RNN, clustering algorithms.

I'm looking for not a solution as such but advice and guidance in the right direction towards solving the problem.

I appreciate any insight you may have to share with me.


texts: Neural networks with not-fixed dimension for input and output

texts:
I would like to know if it exists a model/method which can deal with input and output of different dimension.
For example, let us say that the maximum number of info we could have is 6 features and 5 output.
Then I could have examples with 4 features and 3 output.
Less input features always relates to less output. And relations stays the same.
with only 4 features I have only 4 outputs, and so on.
Most important, it is not that I do not have them for missing knowledge, but because in the same problem dominion I could have all 6 of the features, or less.
It is possibile to create a model which deal with this kind of things ?
The other solution I thought was to just use a simple deep network, with the maximum number of features and output as dimension, and use a value = 0 when I have a missing feature or a missing target. But that destroyed completely the training performances


texts: How to transform one graph to a spectrum?

texts:
Recently, I studied a paper called "What Does Your Chair Know About Your Stress Level?

It can be download at the link below.

  http://www.barnrich.ch/wiki/data/media/pub/2010_what_does_your_chair_know_about_your_stress_level.pdf

at page 210, Fig.5 (picture 5) mentioned that

  Spectra of the norm of the CoP vector during the experiment (stages 3–7) for the same subject used in Fig. 4.

How did they do to transform Fig.4 to Fig.5? And what do x-axis and y-axis mean in Fig.5?

Fig4

Fig5


texts:
For machine learning, we need to normalized the inputs (features) for good results.
Do we need to normalize the output as well during training?


texts: Estimating average daily consumption with samples randomly scattered in time

texts:
I want to estimate my daily water consumption. I have taken pictures of the water meters (total m3 used since last reset) every now and then, but without any regularity. There can be a difference of a few days to several weeks between samples.
What would be the best way to estimate this? I have thought of the following approaches:

Create a double-entry table with the sample dates in the column and in the row headers. Each cell is the average consumption per day between the corresponding two dates. Finally, average all the cells in the table.
Calculate the daily consumption between every two consecutive samples. Finally, average all of them.

It seems to me that the first one would give a better estimation given the higher number of samples compared, but I am not sure if this is valid.


texts: Can Boosted Trees predict below the minimum value of the training label?

texts:
I am using gradient Gradient Boosted Trees (with Catboost) for a Regression task. Can GBtrees predict a label that is below the minimum (or above the max) that was seen in the training ?
For instance if the minimum value the label had is 10, would GBtrees be able to predict 5?


texts: SQL Server credential doesn't work from Orange

texts:
I am trying to have a SQL Table widget in my Orange canvas, which is connected to SQL Server. I have installed pymssql, and in the SQL Table dialog, I do get an option to connect to SQL Server.

However, after entering the credentials, I get the following error

  (20009, b'DB-Lib error message 20009, severity 9:\Unable to connect: Adaptive Server is unavailable or does not exist (DESKTOP-RABJ6QO\MIKA_SQLEXPRESS)\n')

How can I fix it?

Here is a screenshot of the dialog:


texts:
I am using Python/Scikit to do data encoding before I go ahead and train my Neural Network. I have a few columns that look like this

07:05:00
08:41:00
17:25:00
12:58:00
08:56:00
11:59:00
17:25:00
15:24:00

Any suggestions on how to encode this? Or is leaving it like this fine?


texts: What kinds of data other than geographical are topologically spherical?

texts:
I'm trying to think of a data set that is essentially topologically spherical. It's easier to think of cylindrical datasets (two dimensions, one periodic) or toroidal datasets (two dimensions, both periodic).

Obvious candidates are geographical and astronomical, ground and sky; but I think the only thing spherical about the sky is its projection onto the ground, so it really just comes back to Earth.

I find it helpful to think about in terms of a circle's fundamental polygon:


texts: The cross-entropy error function in neural networks

texts:
In the MNIST For ML Beginners they define cross-entropy as

$$H_{y'} (y) := - \sum_{i} y_{i}' \log (y_i)$$

$y_i$ is the predicted probability value for class $i$ and $y_i'$ is the true probability for that class.

Question 1

Isn't it a problem that $y_i$ (in $\log(y_i)$) could be 0? This would mean that we have a really bad classifier, of course. But think of an error in our dataset, e.g. an "obvious" 1 labeled as 3. Would it simply crash? Does the model we chose (softmax activation at the end) basically never give the probability 0 for the correct class?

Question 2

I've learned that cross-entropy is defined as

$$H_{y'}(y) := - \sum_{i} ({y_i' \log(y_i) + (1-y_i') \log (1-y_i)})$$

What is correct? Do you have any textbook references for either version? How do those functions differ in their properties (as error functions for neural networks)?


texts: Tensorflow - I don't get the right shapes - `ValueError: Shapes (9, 1) and (8, 9) are incompatible`

texts:
I want to train a Sequential Neural Net (NN) with Tensorflow.
bidding_nn = tf.keras.Sequential([
tf.keras.layers.Dense(units=128, activation='elu', kernel_initializer='he_uniform'),
#tf.keras.layers.Flatten(),
tf.keras.layers.Dense(units=128, activation='elu', kernel_initializer='he_uniform'),
tf.keras.layers.Dense(units=9, activation='softmax', kernel_initializer='he_uniform'),
])

and
adam = Adam(lr=0.02, decay=0.01)
bidding_nn.compile(optimizer='adam',
          loss=tf.keras.losses.CategoricalCrossentropy(),
          metrics=['accuracy'])

bidding_nn.fit(train_dataset, epochs=10)

It should tell me probabilities for the 9 categories I have.
So as input for the NN, I have 8 npArrays of lengths 32 (one-hot encoded) and as output 1 npArray of lengths 9 (one-hot encoded).
(Pdb) train_dataset
<TensorSliceDataset shapes: ((8, 32), (9,)), types: (tf.float64, tf.float64)>

However, at bidding_nn.fit(train_dataset, epochs=10) I get the error message
    ValueError: Shapes (9, 1) and (8, 9) are incompatible

++++++++++++++++++++++
Answer to the problem, see
https://stackoverflow.com/a/66350991/12368039


texts: How to barplot output of pandas.describe() from multiple datasets


texts:
I am new to D3 programming (any programming, for that matter). I have protein-protein interaction data in JSON format and csv format. I would like to use that data for network visualization.

Data attributes: Protein Name, Protein Group, Protein type, Protein Source Node, Protein Target Node

Can anyone suggest good network visualizations for such data. How does it work with hive plots?


texts: Estimate battery voltage based on scheduled events and previous behaviour

texts:
My goal is to estimate if a battery will have enough charge for certain other systems to be powered. The power state of the other systems is recorded (i.e. if they are turned on or not), as well as the times when the battery is being charged by solar panel during sunlight. These records are several month in the past.

My question is now, in what topic/algorithms should I research when I want to use the previous data and events to estimate/extrapolate the voltage into the future (to make scheduling of powering systems easier, such as canceling them if the battery will be drained too much)? The upcoming charging times can be calculated, and it'll also be known when a system will be powered.

All the datasets (i.e. voltage, charging times, events) are available as timeseries (but with different timestamps/sample rates).


texts: Data leakage in bidirectional LSTM timeseries data


texts: Clearest way to visualise temporal data?

texts:
I've collected bus arrival times at my local bus stop from the past month - so I have every time my bus (a specific bus number) shows up at my bus stop for each day of the week (Monday, Tuesday, etc.).

I am struggling at determining the best and clearest way to display this data. Eventually I want to use a clustering algorithm to help understand the most likely time the bus will show up. So on a Monday I know if the bus is more likely to show up at 7.45am or 7.48 am.

I believe 8 charts will be best - one chart for each day of the week, and then one final chart that shows the average regardless of day of week. What would be my best chart type for clearly visualising this data?


texts: How to preprocess data for image classification from a .txt file?

texts:
Basically my issue is that im building an image classification model using AlexNet. I have this pre-split dataset thats already split into training, test, validation. However the issue is that these splits are in .txt files (e.g. trainingsplit.txt) and inside the .txt file is a list of image_001.png files etc. How do I extract this data and pass it through my preprocessor? This is what I would normally do for preprocessing:

train_dir = '/content/gdrive/My Drive/alexNet/Training'
validation_dir = '/content/gdrive/My Drive/alexNet/Validation'

train_datagen = ImageDataGenerator(
      rescale=1./255,
      rotation_range=40,
      width_shift_range=0.2,
      height_shift_range=0.2,
      shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True,
      fill_mode='nearest',
      )

test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
        train_dir,  # This is the source directory for training images
        target_size=(224, 224),
        batch_size=20,
        class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
        validation_dir,
        target_size=(224, 224),
        batch_size=20,
        class_mode='binary')

Edit: Content of training.txt

image_001.jpg
image_002.jpg
image_014.jpg
image_017.jpg
etc.

The training data is a .txt file containing several rows of text, which are the names of different images. This txt file is inside the main directory Dataset, which contains class directories like Apples, Bananas, Oranges, etc.


texts: Fast way of computing covariance matrix of nonstationary kernel in Python

texts:
Suppose I have symmetric positive definite covariance function $k:\mathbb{R}\times\mathbb{R}\rightarrow \mathbb{R}$ that is non-stationary (i.e. $k(x,y) \neq g(|x-y|)$ for any function $g$). Is there a fast way in Python given design points $(x_1,\ldots,x_n$) to calculate its covariance matrix $(k(x_i,x_j))_{i,j}$?

If the covariance function is stationary then we can compute the whole matrix at once using numpy's matrix operations and avoid slow Python loops - e.g. in this.

Currently my implementation is:

dim = len(X)
kern_mat = np.zeros((dim,dim))
   for i in range(dim):
      for j in range(i+1):
         kern_mat[i,j] = kern(X[i],X[j])
         kern_mat[j,i] = kern_mat[i,j]

Any help with speedups or otherwise is appreciated!


texts: Quantifying 'growth friction' when projecting target goals

texts:
As part of my DS work I spend some fraction of my time helping the team make growth projections, either for setting growth targets or when forecasting actual data.

There is obviously a range of ways to go about doing this but the one thing I don't have a good solution for at the moment is being able to fold in or at least quantify how much harder it is to grow in a market this year as opposed to historical years. It is easy to anchor growth on historical rates but in markets at which you've got a solid penetration already and are perhaps heading toward saturation it is likely a harder task to maintain a historical level of growth than it was to achieve that level the first time around.

I've thought about a few approaches that basically involve taking the derivative of a growth curve to look at the rate of change of growth, but it is hard to know if that is really probing the truth of how much harder it is to grow, or if its just that we're doing a worse job and growing more slowly!

Does anyone have a good approach to either measuring some sort of 'growth friction' for a current year vs historical years, or perhaps some other forecasting technique that allows information about this to be folded in somehow?


texts:
I need some help to understand the meaning between these different scores.
Currently, I am doing the classification problems using machine learning, and I have obtained the results for the classification as shown in the image below.

To obtain the results like in the image I use the code:

from sklearn.metrics import classification_report, confusion_matrix, accuracy_score
print(confusion_matrix(y_test,y_pred))
print(accuracy_score(y_test,y_pred))
print(classification_report(y_test, y_pred))

Then I also try to use the code below to get the recall, precision and f1-score

print(precision_score(y_test,y_pred))
print(recall_score(y_test,y_pred))
print(f1_score(y_test,y_pred))

Results :
Precision : 0.19601
Recall : 0.44360
F1-score : 0.27188

Then I also try this code for weighted using this code:

print(precision_score(y_test,y_pred, average='weighted'))
print(recall_score(y_test,y_pred, average='weighted'))
print(f1_score(y_test,y_pred, average='weighted'))

Results :
Weighted Precision : 0.8588
Weighted Recall : 0.7684
Weighted F1-score : 0.8048

The problem right now is I already confuse with all these values. What does the meaning of avg/total value in the image, the values of the second code I try, and the value of the weighted metrics as my third code. Which value that I should use to know that the performance of the classifier is good or not ? Hope someone can help me.


texts:
so some context, I'm trying to develop an OCR (for fun) and for that reason I decided to first find text within a page, parse it in to letters within the text and from there try and classify the letters that were extracted one by one.

For the classification I'm trying to build a CNN based upon the following database:
http://www.ee.surrey.ac.uk/CVSSP/demos/chars74k/

and the following CNN architecture:

 # CNN architecture
input = Input(shape=(32, 32, 1))
conv1 = Conv2D(32, (3, 3))(input)
pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)
drop1 = Dropout(0.5)(pool1)
conv2 = Conv2D(64, (3, 3))(drop1)
pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)
drop2 = Dropout(0.5)(pool2)
flatten = Flatten()(drop2)
fully_connected = Dense(512, activation='relu')(flatten)
output = Dense(62)(fully_connected)
model = Model(inputs=input, outputs=output)
model.compile(loss='sparse_categorical_crossentropy',optimizer='Adam', metrics=['accuracy'])
print(model.summary())
model.fit(X, y, epochs=10 , validation_split=0.3)

However, all I seen to get no matter what I try (epochs, batch size, validation split) etc is 0 accuracy.... So assuming my database and labels are fine... what could be going wrong?


texts: Instance segmentation mask creation for adjacent objects within the same class

texts:
How are instance segmentation masks created for adjacent objects from the same class ? If for example we have different objects, we can distinguish between them using the labels, but how to distinguish between adjacent objects since they are annotated the same way within the image ?
Thank you in advance !


texts: How do I find the minimum distance between zip codes in R?

texts:
I have a dataset that lists all zip codes in the U.S., their types (standard, po box, university, etc). I want to replace po box and university zip codes with the next closest standard zip code. I broke down the dataset by state so that R wouldn't have to make as many calculations. In theory, I would like to have standard zip codes in the first column and zip codes that need replacement in the first row, and have the distance between the two be the intersection value.

For example,

                REP 1   REP 2   REP 3   REP 4
STD 1           0.215   0.152   0.025   0.124
STD 2           0.365   0.410   0.074   0.234
STD 3           0.234   0.201   1.322   0.683
STD 4           0.543   0.282   0.483   0.094
MINS            STD 1   STD 1   STD 2   STD 4

where STD 1 is a standard zip code with its own latitude and longitude, and REP 1 is a zip code that needs to be replaced (is a university/po box zip) with its own latitude and longitude. I only have about 5 weeks of experience in R, so please bear with me if something doesn't quite make sense to me immediately. I have tried to do this in excel and having a sheet with close to 10,000 columns by 40,000 rows crashes every time that I try to calculate all of the distances because there are just too many calculations.

I have a feeling that either the apply() or mapply() functions are needed here. I want to calculate the distances using a formula that considers the curvature of the earth, (euclidean, etc) like dist() or the geosphere package to maintain accuracy and be reproducible.

If there is anything else that would be helpful to add on here, let me know and I'll upload it asap. Here is my R code for Alaska, the first state in alphabetical order.

AK<-subset(db,STAABBRV.x=="AK")
AKPO<-subset(AK,ZipCodeType!="STANDARD",select=c("ZIP_CODE","ZipCodeType","Long","Lat"))
AKPO<-within(AKPO,{IS_PO=ifelse(ZipCodeType!="STANDARD",1,0)})
AKSTANDARD<-subset(AK,ZipCodeType=="STANDARD",select=c("ZIP_CODE","ZipCodeType","Long","Lat"))
AKSTANDARD<-within(AKSTANDARD,{IS_PO=ifelse(ZipCodeType!="STANDARD",1,0)})
table<-rbind(AKSTANDARD,AKPO)
table$ZipCodeType<-NULL
rm(AK,AKPO,AKSTANDARD)

This sets up a table that has column names "ZIP_CODE", "Long", "Lat", and "IS_PO". "IS_PO" is a numerical indicator for whether or not the zip code is standard or po/university. 1 indicates that the zip code is a po/univ zip and 0 indicates a standard zip. I did this because some functions required that the data in the dataset be the same type (numerical).

Here are some of my failed attempts at writing code to calculate the minimum distances.

 lapply(bit::chunk(1, nrow(zipcode), 1e2), function(ridx) {
  merge(zipcode, zipcode[ridx[1]:ridx[2]], by = "dum", allow.cartesian = T)[
    , dist := distGeo(matrix(c(longitude.x, latitude.x), ncol = 2),
                      matrix(c(longitude.y, latitude.y), ncol = 2))/1609.34 # meters to miles
    ][dist <= 5 # necessary distance treshold
      ][, dum := NULL]
}) %>% rbindlist -> zip_nearby_dt

DOESITWORK<-apply(db, 1, function(x) spDistsN1(matrix(x[3:4], nrow=1),
                                    x[5:6],
                                    longlat=TRUE))

mins<-apply(Lat,1,function(x)return(array(which.min(x))))
mins<-data.frame(row=names(mins),col=mins)
Lat$mins<-apply(mins,1,FUN=function(x)return(paste(x["row"],colnames(Lat[as.numeric(x["col"])]),Lat[x["row"],as.numeric(x["col"])],sep="/")))


texts: GCloud AI Platform for an ETL process is the right approach?

texts:
I'm involved in a project consisting in:

Extracting from the internet a document (file of 8/10 pages)

Processing it (extract text, clean, tokenize and lemmatizing)

Loading it to a Relational data base

The problem is that for the step of lemmatizing takes too long (20' for doc approx) that sums up to not feasible timing as I want to do it for 1000/2000 docs. For that reason I'm considering migrating to google cloud AI Platform using more powerful resources. I've extensively read about the posibilities of AI Platform for training models, but is it also adecuated for my task? How could I do it? Any general advice will be appreciated.


texts:
I'm trying to perform a stacking ensemble of three VGG-16 models, all custom-trained on my personal dataset and having the same input shape. This is the code:

input_shape = (256,256,3)
model_input = Input(shape=input_shape)

def load_all_models(n_models):
    all_models = list()
    model_top1 = load_model('weights/vgg16_1.h5')
    all_models.append(model_top1)
    model_top2 = load_model('weights/vgg16_2.h5')
    all_models.append(model_top2)
    model_top3 = load_model('weights/vgg16_3.h5')
    all_models.append(model_top3)
    return all_models

n_members = 3
members = load_all_models(n_members)
print('Loaded %d models' % len(members))

#perform stacking
def define_stacked_model(members):
    for i in range(len(members)):
        model = members[i]
        for layer in model.layers:
        # make not trainable
            layer.trainable = False
            # rename to avoid 'unique layer name' issue
            layer.name = 'ensemble_' + str(i+1) + '_' + layer.name
    # define multi-headed input
    ensemble_visible = [model.input]
    # concatenate merge output from each model
    ensemble_outputs = [model.output for model in members]
    merge = keras.layers.concatenate(ensemble_outputs)
    hidden = Dense(6, activation='relu')(merge)
    output = Dense(2, activation='softmax')(hidden)
    model = Model(inputs=ensemble_visible, outputs=output)
    # compile
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model

# define ensemble model
stacked_model = define_stacked_model(members)
stacked_model.summary()

The ensemble model is expected to have a multi-headed input and a single output. Upon running the code, I get the graph disconnected error like this:

ValueError: Graph disconnected: cannot obtain value for tensor Tensor("input_2_2:0", shape=(?, 256, 256, 3), dtype=float32) at layer "ensemble_2_input_2". The following previous layers were accessed without issue: ['ensemble_3_input_2']

Kindly help resolve the error.


texts: Storing multiple time series in a python pandas dataframe efficiently

texts:
I got different time series. Each time series contains different attributes, which have data entries at different points of time (timestamps). Each time series is labeled and has a label attribute.
I made a graphic showing the structure of the data.

It is important to me to have a structure which is persistable as a csv. I tried this with nested dataframes, but the nested dataframe is not saved correctly.
The dataset is about 250 Megabytes, stored in a numpy array of dimension (3, 1200, 2501, 10).
I am looking for an efficient approach to create such a dataframe with the given data structure, such that it is persistable with pandas and file size still efficient.
Edit:
The file format does not necessarily need to be csv.


texts: How to implement PCA color augmentation as discussed in AlexNet

texts:
I read through "ImageNet Classification with Deep Convolutional Neural Networks" again specifically for details on how to implement PCA color augmentation.
I am unsure if I have it right. Here is how I did this in numpy:

# original image is 224x224x3 of dtype uint8

renorm_image = np.reshape(original_image,(original_image.shape[0]*original_image.shape[1],3))

renorm_image = renorm_image.astype('float32')
renorm_image -= np.mean(renorm_image, axis=0)
renorm_image /= np.std(renorm_image, axis=0)

cov = np.cov(renorm_image, rowvar=False)

lambdas, p = np.linalg.eig(cov)
alphas = np.random.normal(0, 0.1, 3)

delta = np.dot(p, alphas*lambdas)

delta = (delta*255.).astype('int8')

pca_color_image = np.maximum(np.minimum(original_image + delta, 255), 0).astype('uint8')

One serious doubt is the line "delta = (delta*255.)". I have to do this to rescale things such that the numbers make sense. I hope someone can give me feedback if this is right.


texts: How to compute score and predict for outcome after N days

texts:
Let's say I have a medical dataset/EHR dataset that is retrospective and longitudinal in nature. Meaning one person has multiple measurements across multiple time points (in the past). I did post here but couldn't get any response. So, posting it here
This dataset contains information about patients' diagnosis, mortality flag, labs, admissions, and drugs consumed, etc.
Now, if I would like to find out predictors that can influence mortality, I can use logistic regression (whether the patient will die or not).
But my objective is to find out what are the predictors that can help me predict whether a person will die in the next 30 days or the next 240 days, how can I do this using ML/Data Analysis techniques?
In addition, I would also like to compute a score that can indicate the likelihood that this person will die in the next 30 days? How can I compute the scores? Any tutorials links on how is this score derived?, please?
Can you please let me know what are the different analytic techniques that I can use to address this problem and different approaches to calculate score?
I would like to read and try solving problems like this


texts: Generate series of values using Keras GAN architecture

texts:
I'm trying to generate something like that:
Which is a random sample from my real data function (that i'm trying to mimic).
numbers = [ 54.,  87., 103., 209., 356., 371., 383., 448., 452.,  38.,  37.,
          30.,  22.,  24.,  48.,  52.,  61.,  66.,  87., 150., 163., 406.,
         545., 557., 566., 508., 413.,  26.,  22.,  22.,  12.,  19.,  22.,
          26.,  28.,  90.,  93., 126., 155., 362., 476., 470., 455., 410.,
         345., 252., 233.,  62.,  45.,  42.,  40.,  38.,  37.,  26.,  27.,
          16.,  18.,  21.,  21.,  22.,  63.,  67.,  96., 177., 228., 331.,
         382., 183.,  38.,  31.,  14.,  13.,  13.,  18.,  19.,  21.,  23.,
          61.,  68.,  93., 104., 179., 273., 428., 446., 388.,  96.,  77.,
          19.,  18.,  13.,   4.,   0.,   0.,   0.,   0.,   0.,   0.,   0.,
           0.]

How would your Generator & Discriminator would look like???
Here's my attempt to create G & D that will be able to do so:
def create_G(num=100):
  G_in = Input(shape=num)
  x = Dense(num/2,activation=LeakyReLU())(G_in)
  x = Dropout(0.5)(x)
  x = Dense(num/4,activation=LeakyReLU())(x)
  x = Dropout(0.5)(x)
  x = Dense(num)(x)
  G = Model(G_in,x)
  G.compile(loss='binary_crossentropy',optimizer=Adam(learning_rate=0.001))
  return G

def create_D(num=100):
  D_in = Input(shape=num)
  x = Reshape((-1,1))(D_in)
  x = Conv1D(num,3,activation='relu')(x)
  x = Dropout(0.5)(x)
  x = Flatten()(x)
  x = Dense(2,activation='sigmoid')(x)
  D = Model(D_in,x)
  D.compile(loss='binary_crossentropy',optimizer=Adam(lr=0.003))
  return D

I'm training the right way, with fake and real examples.
But my G is not able to fool the D.
Here is an example of the values of G.predict(noise) in blue, compare to the real values in orange .

What am I missing?
Would appreciate any suggestions for different architecture.


texts: How can we use the current rewards as a system input in the RUN time when working with Deep Q learning?

texts:
When tuning the Deep Q net parameters we use the immediate rewards . Specially in action replay and regressing .

But in the run time we don't care about the rewards at all. Because our neural network will take states as the input . I am explaining this relevant to the deep mind Atari game play paper . Any one thing it is wastage of input ? Is there a mechanism we can use something like Score even in the run time .

p.s - I understand how they update the Neural Net parameters with regression . Sometimes with TD(λ) .


texts:
Encountred an index error when trying to run the code found here.
The error occurs in section In[10]:
for n in range(len(dominantTheta)):
    shift = dominantTheta[n]
    regressionDelta += dominantAmp[n] * np.cos(n * np.array(range(len(df))) + shift)

Error message:
Traceback (most recent call last):
  File "C:/Users/xxx/Desktop/Python Workspace/DSP.py", line 62, in <module>
    shift = dominantTheta[n]
  File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\core\series.py", line 868, in __getitem__
    result = self.index.get_value(self, key)
  File "C:\Users\xxx\AppData\Local\Programs\Python\Python36\lib\site-packages\pandas\core\indexes\base.py", line 4375, in get_value
    tz=getattr(series.dtype, 'tz', None))
  File "pandas\_libs\index.pyx", line 81, in pandas._libs.index.IndexEngine.get_value
  File "pandas\_libs\index.pyx", line 89, in pandas._libs.index.IndexEngine.get_value
  File "pandas\_libs\index.pyx", line 132, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\hashtable_class_helper.pxi", line 987, in pandas._libs.hashtable.Int64HashTable.get_item
  File "pandas\_libs\hashtable_class_helper.pxi", line 993, in pandas._libs.hashtable.Int64HashTable.get_item
KeyError: 0

I have the latest versions of numpy and pandas installed, and using Python 3.6


texts: How to detect multiple handles used by the same individual?

texts:
Suppose a person (John Doe) uses different user names (handles) in different circumstances (for simplicity, let us limit the conversation to a single social network, e.g., twitter). E.g., he

posts pictures of kittens as john_doe,
posts political rants as deplorable_john,
shares technology news as super_doe
&c (and each screen_name has a different id, so no short cut).

How would you go about detecting that all these accounts are used by the same individual?

SNA: look at who they follow (and maybe who follows them): if these sets are sufficiently similar, it's a good signal.
NLP: compare the posted texts and try to detect that they have been written by the same author.
Meta data: see if the accounts are usually active from the same geo location at the same time.

What other approaches have been / could be tried?

Any published work?


texts: create and use weights in Python to perform weighted correlation and PCA

texts:
I need help with the principal components’ analysis code below in 3 ways:
Write Code to create RIM (RAKE) weighting.
I’m trying to ensure that using the code below, I get exactly the same results than in SPSS.
If you change the weights below to 1, the two codes needs to yield the same results (weighting with 1 is the same than no weighting). However the two codes below do not give the same results if you change the weighing to 1 !
Here are the python codes I wrote:
Without weighting
import numpy as np
import pandas as pd
import math
import statistics
from sklearn.decomposition
from sklearn.preprocessing import StandardScaler

d = {
'gender':[0,0,1,1,1,1,1,1],
'v1':[17,18,11,16,11,18,15,19],
'v2':[2.54,2.71,2.07,2.50,1.96,2.17,1.63,2.36],
'v3':[2.02,2.47,2.12,2.53,1.50,2.39,2.45,2.02],
'v4':[2.34,2.08,1.49,2.05,1.70,2.42,1.68,2.21]
    }
df = pd.DataFrame(data=d)

x = StandardScaler().fit_transform(df.loc[:, ['v1','v2','v3','v4']])
cor_mat1 = np.corrcoef(x.T)
eig_vals, eig_vecs = np.linalg.eig(cor_mat1)
eig_pairs = [(np.abs(eig_vals[i]), eig_vecs[:,i]) for i in range(len(eig_vals))]
eig_pairs.sort(key=lambda x: x[0], reverse=True)
FAC1_1 = x.dot(eig_pairs[0][1].reshape(4,1))

print(FAC1_1)

**With weighting**

import numpy as np
import pandas as pd
import math
import statistics
from sklearn.preprocessing import StandardScaler

def m(x, w):
    return np.sum(x * w) / np.sum(w)
def cov(x,y, w):
    return np.sum(w * (x - m(x, w)) * (y - m(y, w))) / np.sum(w)
def corr(df, w):
    dt = pd.DataFrame(np.nan, index=df.columns, columns=df.columns)
    for i in range(df.shape[1]-1):
        dt.iloc[i , i]=1
        for j in range(1,df.shape[1]):
            x=cov(df.iloc[: , i], df.iloc[: , j], w) / np.sqrt(cov(df.iloc[: , i],df.iloc[: , i], w) * cov(df.iloc[: , j],df.iloc[: , j], w))
            dt.iloc[i , j]=x
            dt.iloc[j , i]=x
    dt.iloc[df.shape[1]-1 , df.shape[1]-1]=1
    return dt

d = {
'gender':[0,0,1,1,1,1,1,1],
'v1':[17,18,11,16,11,18,15,19],
'v2':[2.54,2.71,2.07,2.50,1.96,2.17,1.63,2.36],
'v3':[2.02,2.47,2.12,2.53,1.50,2.39,2.45,2.02],
'v4':[2.34,2.08,1.49,2.05,1.70,2.42,1.68,2.21],
'weight':[2,2,0.666666667,0.666666667,0.666666667,0.666666667,0.666666667,0.666666667]
    }
df = pd.DataFrame(data=d)
x = pd.DataFrame(df.loc[:, ['v1','v2','v3','v4']])

cor_mat1 = corr(x,df['weight'])
eig_vals, eig_vecs = np.linalg.eig(cor_mat1)
eig_pairs = [(np.abs(eig_vals[i]), eig_vecs[:,i]) for i in range(len(eig_vals))]
eig_pairs.sort(key=lambda x: x[0], reverse=True)
FAC1_1 = x.dot(eig_pairs[0][1].reshape(4,1))

print(FAC1_1)

Here is the SPSS syntax:
FACTOR
  /VARIABLES v1 v2 v3 v4
  /MISSING LISTWISE
  /ANALYSIS v1 v2 v3 v4
  /PRINT INITIAL EXTRACTION
  /CRITERIA MINEIGEN(1) ITERATE(25)
  /EXTRACTION PC
  /ROTATION NOROTATE
  /SAVE REG(ALL)
  /METHOD=CORRELATION.


texts: Which recommender system: Content based or Collaborative filtering?

texts:
I want to build a recommender system for a coupons website which should do the following:
Given the past purchase behaviour of a user, recommend coupons which the user is likely to buy.
The data does not have any ratings for coupons by the user. It tells if a user bought a certain coupon or not and what is the gender and location of the user who bought these coupons. Between content-based and collborative-filtering what would be the best recommender system to choose from in this scenario. Personally, I think that content based recommender system fits the requirement because I want to look only at user's own historical behaviour. What would you guys advice? Thanks.


texts: Need help with python code as part of a data analysis project

texts:
I am new to python programming. As part of a data analysis project, I am trying to get a scatter plot of Salary vs Wins for each year of 4 consecutive years (so I am trying to get 4 different scatter plots, one for each year). I am using the following code:

teamName = 'OAK'
years = np.arange(2000,2004)
for yr in years:
    df = joined[joined['yearID']==yr]
    plt.scatter(df['salary']/1e6,df['W'])
    plt.title('Wins vs Salaries in year' + str(yr))
    plt.xlabel('Total Salary (in millions)')
    plt.ylabel('Wins')
    plt.xlim(0,180)
    plt.ylim(30,130)
    plt.grid()
    plt.show()

However, I am only getting one plot corresponding to 2003.

Can anyone point out the mistake ?

Thanks


texts:
In the biology class we've learned, that the neurons are connected. And if two or more neurons interact with each other often, then the connection gets stronger and stronger, and new connections may form. But if a connection is unused for a while, then it gets weaker and weaker, and it may lost forever. That is how the human brain/memory learns and forgets something.

I am a bit confused now. Because I've started to learn about artificial neurons, and the method of learning is very different. We analyze, how efficient the network was, and before restarting it, we update the weights a bit, based on the relative difference between the returned and the expected value.

So my question is: How is a real-world brain capable of learning? And does it have a default configuration?


texts: How to avoid covariate shift in python and distribute classes in each train and test phase?

texts:
We all know that with the use of sklearn package from python, we can create X_train, X_test, y_train and y_test via this code:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)

I want to make sure that each training and testing phase of a multi-class data set, have 66.33/33.33 percent of each class values so the prediction and accuracy would get better. All i want is 66.33 percent of class A in training set and 33.33 percent of Class A in test set. And, so on for other classes, like B, C, D and etc. in a given multi-class data set.

Is the code provided enough to achieve this or should i write extra code?

Thanks


texts:
Is it possible to train a LIME explainer for a binary classfier on a dataset without labels?
I need to understand what is the value of storing a  LIME explainer object trained on the same data used to train the model.
In general, does it make sense to keep a trained LIME explainer around to generate explanations during production or is it better to train the LIME explainer on production data whenever is needed?
Another question. If I train a LIME explainer on training data and I use it with test data, does the LIME explainer suffers from data shift?


texts:
In Rapidminer, are the decision tree's weights a measure of the "importance" of attributes in the splitting procedure ?

If yes, why is useful to know these weights ?
Are there better methods to know the most discriminant features in a data set ?


texts: Sort a data frame when column name is assigned in a variable

texts:
I am new to R and hence my question is likely to be basic. I tried researching the answer before posting here however didn't get the answer I was looking for.

I am trying to order a data frame on the basis of certain columns. The column is dynamic and depends on a certain "outcome" which would be from a vector of "validoutcomes".

validoutcomes <- c("A","B","C")

if outcome is "A" then I am supposed to read data from column 11. For "B" its columns 17 and for "C" it is 23.

I renamed the columns so that it is easier for my readability and alignment with  validoutcomes. "input" is my source data from the CSV.

colnames(input)[11] <- validoutcomes[1]
 colnames(input)[17] <- validoutcomes[2]
 colnames(input)[23] <- validoutcomes[3]

I then tried to sort using

statelist <- statelist[order(statelist$outcome)]

Any ideas?


texts: What is the suggested way to create features (Mel-Spectograms) from speech signal for classification with ResNet?

texts:
At the moment I have this piece of code which cuts a Spectogram into fixed length tensors:
def chunks(l, n):
    """Yield successive n-sized chunks from l."""
    for i in range(0, len(l[0][0]), n):
        if(i+n < len(l[0][0])):
            yield X_sample.narrow(2, i, n)

The following piece of code

downsamples the Audio
Creates Mel_Spectograms and takes the log of it
Applies a Cepstral Mean and Variance Normalization
Then it cuts the spectogram with the code above into a fixed size of length and appends it to an array

for index, row in df.iterrows():
    #resample
    wave_form, sample_rate = torchaudio.load(row["path"], normalization=True)
    downsample_resample = torchaudio.transforms.Resample(
    sample_rate, downsample_rate, resampling_method='sinc_interpolation')
    wav = downsample_resample(wave_form)
    mel = torchaudio.transforms.MelSpectrogram(downsample_rate)(wav)
    mellog = np.log(mel + 1e-9)
    X_sample = speechpy.processing.cmvnw(mellog.squeeze(), win_size=301, variance_normalization=True)
    X_sample = torch.tensor(X_sample).unsqueeze(0)
    _min = min(np.amin(X_sample.numpy()),_min)
    _max = max(np.amax(X_sample.numpy()),_max)
    for chunked_X_sample in list(chunks(X_sample,  max_total_context)):
        print(len(chunked_X_sample[0][0]))
        if len(chunked_X_sample[0][0]) == max_total_context:
            X.append(chunked_X_sample)
            y.append(row["y"])

My question: Is this the common way to create features for deep learning?
Do you have any suggestions to optimize this code?
Furthermore I'm not sure if it is right to split the melspectograms instead of splitting the audio earlier.


texts: How to set the parameters of a Hidden Markov Model that'll be used to correct the mistakes by a previous classifier?

texts:
Say we've previously used a neural network or some other classifier C with $N$ training samples $I:=\{I_1,...I_N\}$ (that has a sequence or context, but is ignored by C) the, belonging to $K$ classes. Assume, for some reason (probably some training problem or declaring classes), C is confused and doesn't perform well. The way we assign a class using C to each test data $I$ is: $class(I):= arg max _{ {1 \leq j \leq K} }    p_j(I)$, where $p_j(I)$ is the probability estimate of $I$ corresponding to the $j$-th class, given by C.

Now, on top of this previous classifier C, I'd like to use a Hidden Markov Model (HMM) to "correct" the mistakes made by the previous context-free classifier C, by taking into account the contextual/sequential information not used by C.

Hence let in my HMM, the hidden state $Z_i$ denote the true class of the $i$-th sample $I_i$, and $X_i$ be the predicted class by C. My question is: how could we use the probabilistic information $cl(I):= arg max _{ {1 \leq j \leq K} }    p_j(I)$ to train this HMM? I understand that the confusion matrix of C can be used to define the emission prob. of the HMM, but how do we define the transition and start/prior prob.? I'm tempted to define the start/prior prob. vector as $\pi:=(p_1(x_1), ..., p_K(x_1))$.  But I may be wrong. This is my main question.

A follow up question: One can define an HMM in the above way (using confusion matrix and the prob. information from C); call the resulting parameter set $\Theta_0$. But after doing so, is it advisable to estimate the parameters to best fit the data $I$ used for C, while initializing a parameter set with the values mentioned in the previous paragraph?


texts: Why does keras.Model subclass behave differently to the sequential API?

texts:
I have implemented WGAN based on this blog. I tried to implement define_critic() as a subclass of keras.Model with the goal of having the same result.
class CNNBlock(layers.Layer):

    def __init__(self, channels, kernel_size=(4, 4)):
        super(CNNBlock, self).__init__()

        w_init = initializers.RandomNormal(stddev=0.02)
        const = ClipConstraint(0.01)

        self.conv = layers.Conv2D(channels, kernel_size, strides=(2, 2), padding="same",
                                  kernel_initializer=w_init, kernel_constraint=const)
        self.bn = layers.BatchNormalization()
        self.relu = layers.LeakyReLU(alpha=0.2)

    def call(self, input_tensor, training=False):
        x = self.conv(input_tensor)
        x = self.bn(x)
        return self.relu(x)

class CriticModel(Model):

    def __init__(self, channel):
        super(CriticModel, self).__init__()
        self.cnn1 = CNNBlock(channel[0])
        self.cnn2 = CNNBlock(channel[1])

    def call(self, input_tensor, training=False, mask=None):
        x = self.cnn1(input_tensor, training=training)
        x = self.cnn2(x, training=training)
        x = layers.Flatten()(x)
        x = layers.Dense(1)(x)
        return x

    def get_config(self):
        super(CriticModel, self).get_config()

However, when I run the code, I get an error on the call c_model.train_on_batch(X_real, y_real).
Error
ValueError: tf.function-decorated function tried to create variables on non-first call.

I don't understand why my change leads to this error.
The entire code can be found here


texts:
I would like to set up a server which could support a data science team in the following way: be a central point for storing, versioning, sharing and possible also executing Jupyter notebooks.

Some desired properties:

Different users can access the server and open and execute notebooks that were stored by them or by other team members. The interesting question here is what would be the behavior if user X executes cells in a notebook authored by user Y. I guess the notebook should NOT be changed:
Solution should be self-hosted.
Notebooks should be stored either on the server or on Google drive or on self-hosted instance of owncloud.
(Bonus) Notebooks will be under git versioning control (git may be self-hosted. Cannot be bounded to GitHub or something of that sort).

I looked into JupyterHub and Binder. With the former, I didn't understand how to allow cross users access. The latter seems to only support GitHub as the storage of the notebooks.

Do you have experience with either of the solutions?


texts: How does tree-based algorithms handle linearly combined features?

texts:
While I am aware that tree-based algorithms (e.g., DT, RF, XGBoost) are 'immune' to multi-collinearity, how do they handle linearly combined features?
For example, is there is any additional value or harm in including the three feature: a, b and a+b in the model?


texts: Keras model predicts differently when predicting based on video frame vs. individual snapshot

texts:
I'm training a binary image classification model in Keras, with the end goal of parsing through entire videos. I have a script that reads in a few snapshots from training videos and lets the model predict on them. I have another script that reads in a video and runs the model through each individual frame and saves the output to a DataFrame. For some reason, even though many of the images that the image prediction script goes through are pulled straight from a video, the video prediction script will predict completely differently, even though the input image is the same. The video prediction script always predicts the same class for every frame, even for frames that were in the training set.

Here are the prediction parts of my scripts:

#Image Prediction
        image = cv2.resize(cv2.imread(impath), (300, 300))
        if("y" in gray.lower()):
            image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
        else:
            image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
        image = image.astype("float") / 255.0
        image = img_to_array(image)
        image = np.expand_dims(image, axis=0)

        prediction = model.predict(image)[0]

        print(prediction)

#Video Prediction
while success:
        success, frame = self.video.read()
        if(not success):
            break
        image = cv2.resize(frame, (300, 300))
        image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
        image = image.astype("float") / 255.0

        image = img_to_array(image)
        image = np.expand_dims(image, axis=0)

        prediction = self.model.predict(image)[0]

        if(prediction < .5):
            prediction = 0
        else:
            prediction = 1
        print(prediction)

These two scripts preprocess the images identically, there shouldn't be any reason why the predictions would turn out differently. Using np.array_equal I've deduced that the arrays are not different in size, shape, nor content. I've also used imshow to compare and I can confirm that the two are visually identical.

When I print the prediction values during the video prediction, they're all very close to .5, which shouldn't happen very often, because my loss value should be around .4, which means I should be getting way more .9s and .1s than .5s.

I'm open to any and all ideas. Thank you for your help.


texts: "Invalid value" in RMSprop implementation from scratch in Python

texts:
Edit 2: The regularization term (reg_term) is sometimes negative due negatative parameters. Hence S[f"dW{l}"] contains some negative values. I realize the reg_term has to be added before taking the sqrt, like this:
S[f"dW{l}"] = beta2 * S[f"dW{l}"] + (1 - beta2) * (np.square(gradients[f"dW{l}"] + reg_term))

Edit 1: I see that S[f"dW{l}"] contains some negative values. How is this possible when np.square(gradients[f"dW{l}"] always contains positive values?
I have implemented a neural network from scratch which uses mini-batch gradient descent. The network works well. Unfortunately, I can't get my RMSprop implementation to work. I have verified that the network works well with momentum.
I get a RuntimeWarning when training the network with RMSprop: "invalid value encountered in sqrt". This happens in the "RMSprop" update step.
My implementation of update parameters:
def update_parameters(parameters, gradients, V, S, batch_size, t, learning_rate, reg_param):
    L = len(parameters) // 2
    beta1 = 0.9
    beta2 = 0.999
    epsilon = 1e-8

    for l in range(1, L+1):
        reg_term = (reg_param / batch_size) * parameters[f"W{l}"]

        # RMSprop gradients
        S[f"dW{l}"] = beta2 * S[f"dW{l}"] + (1 - beta2) * (np.square(gradients[f"dW{l}"]) + reg_term)
        S[f"db{l}"] = beta2 * S[f"db{l}"] + (1 - beta2) * np.square(gradients[f"db{l}"])

        # RMSprop update
        parameters[f"W{l}"] -= learning_rate * (gradients[f"dW{l}"] / (np.sqrt(S[f"dW{l}"])) + epsilon)
        parameters[f"b{l}"] -= learning_rate * (gradients[f"db{l}"] / (np.sqrt(S[f"db{l}"])) + epsilon)

This is how I initialize the parameters:
def init_params_V_and_S(activation_layers):
    params = {}
    V = {}
    S = {}
    L = len(activation_layers)

    for l in range(1, L):
        params[f"W{l}"] = np.random.randn(activation_layers[l], activation_layers[l-1]) * np.sqrt(2 / activation_layers[l-1])
        params[f"b{l}"] = np.zeros((activation_layers[l], 1))

        # RMSprop params
        S[f"dW{l}"] = np.zeros((activation_layers[l], activation_layers[l-1]))
        S[f"db{l}"] = np.zeros((activation_layers[l], 1))

    return params, V, S

Any ideas what's causing this?


texts: Owen Zhang's slides: what does the "time" mean?

texts: Here is one of Owen Zhang's slideshares. At page 12, what does the time mean?


texts:
I am using Weka Time Series Forecasting to forecast the trend of the topic NLP in 2018. For that I used evaluate on training option in Weka to obtain Root Mean Squared Error (RMSE). The RMSE value I obtain is 0.785.

What does this imply?
Is there a threshold value (e.g., RMSE < 0.2) that we can consider to
accept/reject this prediction?


texts: statistical significance test between binary label features

texts:
I have 667 features and I want to find features that have a significant boundary between a binary class label before I apply a classification model (e.g Naive Bayes/ SVM) to improve classification model learning rate.

What I know is, if the features' values between the two classes are overlapping, this will cause poor classification.

Hence, I have done a 2 samples t-test to calculate the statistical significance of features between binary class label.

from scipy import stats
p=[]
failure = [1]

#separate out the non-failure and failure group data into 2 dataframes to calculate the
t-test between each feature.

df_failure= df.loc[df['label'].isin(failure)]
df_nonfailure= df.loc[~df['label'].isin(failure)]

for x in listofname:

     p.append((stats.ttest_ind(df_failure[x],df_nonfailure[x], equal_var=False)))

My question is, is this a good features selection approach beside recursive feature elimination/wrapper method? Is there any similar method out there?


texts: Is it wrong if I cluster numerical attributes and categorical attributes separately?

texts:
I have a dataset of credit customers containing mixed data types (numerical and categorical with several levels). I am trying to perform segmentation so that I can end up with k groups and then build definitions (based on attributes I have).

While there are solutions for clustering data with mixed data types (K-prototypes, hierarchical clustering with Gower's distance), why would it be wrong to cluster numerical attributes and categorical attributes separately and come up with definitions individually?


texts: Understanding convolutional pooling sizes (deep learning)

texts:
I'm dumb but still trying to understand the code provided from this e-book on deep learning, but it doesn't explain where the n_in=40*4*4 comes from. 40 is from the 40 previous feature maps, but what about the 4*4?

>>> net = Network([
        ConvPoolLayer(image_shape=(mini_batch_size, 1, 28, 28),
                      filter_shape=(20, 1, 5, 5),
                      poolsize=(2, 2),
                      activation_fn=ReLU),
        ConvPoolLayer(image_shape=(mini_batch_size, 20, 12, 12),
                      filter_shape=(40, 20, 5, 5),
                      poolsize=(2, 2),
                      activation_fn=ReLU),
        FullyConnectedLayer(
            n_in=40*4*4, n_out=1000, activation_fn=ReLU, p_dropout=0.5),
        FullyConnectedLayer(
            n_in=1000, n_out=1000, activation_fn=ReLU, p_dropout=0.5),
        SoftmaxLayer(n_in=1000, n_out=10, p_dropout=0.5)],
        mini_batch_size)
>>> net.SGD(expanded_training_data, 40, mini_batch_size, 0.03,
            validation_data, test_data)

For instance, what if I do a similar analysis in 1D as shown below, which should that n_in term be?

>>> net = Network([
        ConvPoolLayer(image_shape=(mini_batch_size, 1, 81, 1),
                      filter_shape=(20, 1, 5, 1),
                      poolsize=(2, 1),
                      activation_fn=ReLU),
        ConvPoolLayer(image_shape=(mini_batch_size, 20, 12, 1),
                      filter_shape=(40, 20, 5, 1),
                      poolsize=(2, 1),
                      activation_fn=ReLU),
        FullyConnectedLayer(
            n_in=40*???, n_out=1000, activation_fn=ReLU, p_dropout=0.5),
        FullyConnectedLayer(
            n_in=1000, n_out=1000, activation_fn=ReLU, p_dropout=0.5),
        SoftmaxLayer(n_in=1000, n_out=10, p_dropout=0.5)],
        mini_batch_size)
>>> net.SGD(expanded_training_data, 40, mini_batch_size, 0.03,
            validation_data, test_data)

Thanks!


texts: What kind of research can be done with genomic data?

texts:
It is well known that science has given us large amounts of free accessible data, such as https://www.1000genomes.org and https://www.ncbi.nlm.nih.gov/genbank. How can we play around with the data and apply data science/machine learning to it? What could be some ideas?

My own ideas:

Biological data visualisation
Gene prediction using hidden-markov-model

Any more?


texts: Random Forest: how efficient is it for features in a range of values (not binary thresholding)?

texts:
Is it correct that each feature is only used once in one decision tree ? Then, if a feature (the most decisive for the classification) determines a classification for a range of values, can the algorithm detect that in any way or will it fail ?
To be more clear, let's say the classification is to classify people as teenager or not. Then the determinant feature will be the age, and classification will be if this feature is between 12 and 18.
So we don't have only one threshold (age<12) or (age>18), but two.
Can the Random Forest algorithm succeed in this case ?
Thanks for any suggestion


texts:
Is the above graph a white noise?
I'm confused by the spikes at certain places.
The above plot has been obtained after doing a first order differencing on a time series. How can I justify whether there is seasonality present/absent in the data after differencing?


texts: How to train a classification algorithm with normalized data set using scikit-learn python

texts:
I have a dataset in CSV format, 6 columns and 1877 rows. The full dataset can be viewed at ShareCSV.

The first five columns are characteristics and the final column is a binary result, I want to create a classification network to predict result using the five inputs as seen in the CSV above.

I use the following code to normalize the data with pandas, then split into testing and training groups.

from sklearn import preprocessing
import pandas as pd

df = pd.read_csv(r"D:\path\data.csv", sep=",")

df=(df-df.min())/(df.max()-df.min())

X = df[["Title words", "Hour posted", "Author age", "Is Text", "Subreddit age"]]

Y = df['Front']

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2, random_state=42)

I now need to pass this data to scikit-learn and select a classification algorithm, however this is where I am unsure what would be optimal, if anyone could recommend the best algorithm for my data and a rough implementation that would be great.


texts: What are references explaining Hugo Steinhaus early "data science" work?

texts:
Historical background:  Hugo Steinhaus can be considered as an early father of data science. He authored the paper Sur la division des corps matériels en parties, Bulletin de l’Académie Polonaise des Sciences, 1956. The title could read "On the division of material bodies into parts" in English. As the original paper is hard to obtain, the above link is a retranscription from  latex that I made in the summer.

It is considered as one of the first known origins of the $K$-means algorithms (see Data Clustering: 50 Years Beyond K-Means, Anil K. Jain, Pattern recognition letters, 2010 or Origins and extensions of the k-means algorithm in cluster analysis, Hans-Hermann Bock, Electronic Journ@l for History of Probability and Statistics, 2008).

According to his wikipedia page, H. Steinhaus  has collaborated with engineers, geologists, economists, physicians, biologists (and even lawyers), and lacking of trustworthy information during World War II, he invented a statistical tool to estimate 'German' losses, using necrologic news from German soldiers on the front. He notably used the mention that the soldier killed was the first, second or third child from a family.

I am looking for references (possibly second-hand notes, translated papers) to this applied and "missing value" statistical tool related works? Namely:

Regarding his war lacunary inferences, where is this algorithm described (as I would like to be able to re-implement it)?
What are specific examples of his applied works with biologists for instance, were they published, and where?


texts: Neural network output is 0 for test data (using RELU for activation)

texts:
Maybe this is a naive question, but I have a NN that uses relu for all layers. In train data there is no problem, but in test (or validation) the outputs are all 0. I used MinMaxScaler with a feature range of (0,1) and in the test data there are actually negative values that are given as inputs to the NN. This is because I applied the MinMaxScaler only to the train data and then applied the scaler's transform function to the test data, resulting in negative values.

Are those negative values the reason because I'm getting 0 as outputs? How can I correct this?
The way to apply the MinMaxScaler is correct, right? Do fit_transform only to train data and just transform to validation data?
Is best practice to use StandarScaler or MinMaxScaler should be the way?

Thank you!
EDIT: If it is useful, what I'm trying is to get as output is a number greater than 0. It is a regression problem.


texts: Calculation of distance between samples in data mining

texts: I am confused about a little issue related to distance calculation. What I want to know is, while calculating the distance between samples in classification or regression, is the label or output class also used or the distance is calculated using all other attributes excluding the label attribute?


texts:
As far as I understand (pardon me if I am wrong) the activation functions in a neural network go through the following transformations:

Multiplication by constants(weights) to x ( $f(ax)$ , $f(x)$ being the activation function).
Recursive substitution $f(f(x))$.

Now with the above transformations a ReLU activation function should never be able to fit a x² curve. It can approximate, but as the input grows the error of that approximated function will also grow exponentially, right?

Now x² is a simple curve. How can ReLU perform better for real data which will be way more complicated than x²?

I am new to machine learning. So please pardon me if there are any blunders in anything I am assuming.


texts: How to perform tokenization for tweets in xlnet?

texts:
X_train has only one column that contains all tweets.
xlnet_model = 'xlnet-large-cased'
xlnet_tokenizer = XLNetTokenizer.from_pretrained(xlnet_model)

def get_inputs(tweets, tokenizer, max_len=120):
    """ Gets tensors from text using the tokenizer provided"""
    inps = [tokenizer.encode_plus(t, max_length=max_len, pad_to_max_length=True, add_special_tokens=True) for t in tweets]
    inp_tok = np.array([a['input_ids'] for a in inps])
    ids = np.array([a['attention_mask'] for a in inps])
    segments = np.array([a['token_type_ids'] for a in inps])
    return inp_tok, ids, segments

inp_tok, ids, segments = get_inputs(X_train, xlnet_tokenizer)

AttributeError: 'NoneType' object has no attribute 'encode_plus'


texts:
I am working with the titanic dataset and using decision trees for analyzing the age covariate. I'd like just to see whether kids are more likely to survive than adults. I implemented my own Gini coefficient and I had plot the coefficient by age: dataset here titanic ds

import pandas as pd
import seaborn
import matplotlib.pyplot as plt
from sklearn import tree
import graphviz
import numpy as np

def gini_by_age(df, t):
    df['age_group'] = df['age'].apply(lambda row : 0 if row <= t else 1)
    kids = df[df['age_group'] == 0]
    kids0 = kids[kids['survived'] == 0]
    kids1 = kids[kids['survived'] == 1]
    adults = df[df['age_group'] == 1]
    adults0 = adults[adults['survived'] == 0]
    adults1 = adults[adults['survived'] == 1]
    gk = 1 - (len(kids0)**2 + len(kids1)**2)/float(len(kids))**2
    ga = 1 - (len(adults0)**2 + len(adults1)**2)/float(len(adults))**2

    return gk + ga

def plot_gini_by_age(df):
    ages = range(2,25)
    y = [gini_by_age(df, a) for a in ages]
    plt.plot(ages, y)
    plt.show()

def use_tree(df):
    X = np.array(df['age']).reshape((len(df['age']),1))
    y = df['survived']
    clf = tree.DecisionTreeClassifier(max_depth=1).fit(X,y)
    dot_data = tree.export_graphviz(clf, out_file=None)
    graph = graphviz.Source(dot_data)
    graph.render("age")

titanic_df = pd.read_csv("titanic_ds.csv")
ages_cov = titanic_df[['age', 'survived']].dropna()
plot_gini_by_age(ages_cov)
use_tree(ages_cov)
print gini_by_age(ages_cov, 5)
print gini_by_age(ages_cov, 8.5)
print gini_by_age(ages_cov, 15)

output:
    0.925844132419
    0.937732003001
    0.963875889772
I see from the plot that gini coefficient has local minima at roughly 5, 8 and 15 years old and the best is at 5. But scikit gives me 8.5 years old as the best split. What is wrong here?


texts: How to reduce the Root mean square error

texts:
I have dataset which describe "how many passenger arriving in some airport " and I would like to predict how many passenger arriving in monthly bases for next year. The features that I have is the following :

year,
month,
airport,
number of passenger (monthly)

In the data, three out of of 50 airports are usually have huge passengers arriving.

I have used random forest classifier but the issue that i'm encounter is I have RMSE is high. As a result i see huge different between the actual value and predicting value. How to fix this issue ?


texts: How to set class weights for imbalanced classes in Tensorflow?

texts: Do we have an equivalence for the following question, but with tensorflow : How to set class weights for imbalanced classes in Keras? ??


texts: Pyspark: Filter dataframe based on separate specific conditions

texts:
How can I select only certain entries that match my condition and from those entries, filter again using regex?

For instance, I have this dataframe (df)

+----+----+----+----+
|col1|col2|col3|col4|
+----+----+----+----+
|   A|   f|   5|   g|
|   D|  er|  2e|  sd|
|   F|   g|23sd|   a|
|   F| fgf|  45|   d|
|   E|   r|   3|   e|
|   A|  sd|  8f|  dw|
|   F|  sd| 3h1|   d|
+----+----+----+----+

I want to select those entries with 'F' value in col1, and filter again with regex ([a-zA-Z0-9]+) to get only entries with numbers and letters.

+----+----+----+----+         +----+----+----+----+
|col1|col2|col3|col4|         |col1|col2|col3|col4|
+----+----+----+----+         +----+----+----+----+
|   F|   g|23sd|   a|   -->   |   F|   g|23sd|   a|
|   F| fgf|  45|   d|         |   F|  sd| 3h1|   d|
|   F|  sd| 3h1|   d|         +----+----+----+----+
+----+----+----+----+


texts: General approach on time series for customer retention/churn in retail

texts:
I have a time series of data in the following form:

| purchase_date    |    cutomer_id  |   num_purchases | churned |
   2018-10-31            id1              39             0
   2018-11-31            id1              0              0
   2019-01-31            id1              6              1
   2019-03-31            id2              300            0
   2018-04-31            id2               2             1
    ...

I grouped the data by month and summed num_purchases by month. The churned column for user id1 for example represents in which month customer churned. So id1 in my case churned in January. Before this, to label who has churned or not, we sampled customers based on 2 months of inactivity  period from the churn date. I need to predict if a user is going to churn in a 2 months from now. I am not sure what is the best approach for this.

Q1: Should I be grouping customers like I am doing, on a monthly basis or I have to group them on a 2-month basis since that is how they were labeled?
Q2: Also, how do I model this? Do I keep customer_id as a feature of the model or not? Is the gap in dates for each customer relevant and how should I deal with it (if)? The dates repeat for different users, should I create index out of a date but it won't be unique or should I create index out of customer_id?
Q3: If I need to predict whether the user is going to churn by the end of the year for example or in the next 6 months, would that change how I group/arrange my date and model this?

I plan to add more features to this dataframe (both categorical and numerical).


texts: Is there an asymmetric version of nominal correlation?

texts:
I use Cramer's V to calculate correlation of features in a dataset made of only nominal features.

Let's consider the following dataset:

a  |  b
--------
0  |  0
0  |  1
0  |  0
1  |  2
1  |  2
1  |  3

Calculating Cramer's V for features a and b yields 0.707. Since it's symmetric, there's information loss in this case - as we can see, knowing the value of b means we know for sure what is the value of a, but this is no the case if we are given the value of a; in this case, the number possible values of b decreases, but it's still not known for sure.

I'd like to find an asymmetric metric that will provide this information for nominal values - meaning, will give a different value when calculated a -> b and b -> a. Is there anything like this?


texts: Why is the behaviour of groupby command different in R and in python with pandas?

texts:
if I execute the following lines of codes in R as well as python for groupby command, the order of the resulting dataframes are not matching when I convert them to dataframes after that. How to maintain the order to be same when I execute the R code in Python.
in R
project_list <- project_data %>%
  group_by(entity)%>%
  distinct(Manager)
############
Python
project_list=project_data.groupby('entity').Manager.unique()
project_list=pd.DataFrame(project_list)
project_list=project_list.reset_index()
project_list['Manager']=project_list['Manager'].str.get(0)
#############
in R                in Python
entity   Manager  entity   Manager
ML_10     A        ML_25     D
ML_08     B        ML_10     A
ML_25     D        ML_08     B
But the expected results should be similar to the one from R

What change need to be done in Python code to get the same results?


texts: How to define the adequate cash prize sizing for hosting a Kaggle or similar compeition?

texts:
If you take Kaggle as a well known example of data science competition size, how do you know what is an adequate budget for the cash prize size? At least to determine an order of magnitude, given I am not able to study all previous competitions, cluster them by this factor and assess driving factors? irony - could be a competition itself?

I've also looked up the Q&A page at Kaggle and have found more or less same question but without answer yet:

We would like to understand:

  What is the minimum prize pool that starts capturing your attention?

  ..

  Open-ended question - could you give us idea on what are the most important things you take into account before deciding to participate
  are not?


texts: Which is the best way to select categorical features with Autoencoders in Python?

texts:
I have a dataset containing both categorical and numerical features. I am trying to work with Autoencoders for feature selection, so the first thing I do is to normalise the numerical features. For categorical features, I would like to apply categorical embedding, instead of Label Encoder or One Hot Encoder.
As the categorical embedding method will create N different variables to represent each categorical feature, the Autoencoder will choose one or many of those variables as selected features.
Thus, how could be the best way to face this situation?


texts:
I am new to Machine Learning and Data Science. By spending some time online, I was able to understand the perceptron learning rule fairly well. But I am still clueless about how to apply it to a set of data. For example we may have the following values of $x_1$, $x_2$ and $d$ respectively:-

\begin{align}&(0.6 , 0.9 , 0)\\
&(-0.9 , 1.7 , 1)\\
&(0.1 , 1.4 , 1)\\
&(1.2 , 0.9 , 0)\end{align}

I can't think of how to begin.

I think we need to follow these rules.

$$W_i = W_i + \Delta W_i$$
$$\Delta W_i = \eta(d_i - y_i)$$
$$\text{ If} y_i = \sum w_ix_i \ge 0, y = 1$ \text{ else} y=0$$
$$x_0 (\text{Bias}) = 0 $$

Where $d_i$ is the target value, $y_i$ is the output value
$\eta$ is the learning rate and $x_i$ is the input value

Any help is appreciated. Thanks!


texts:
I keep receiving the error message "The kernel appears to have died" when running a simple command such as pd.read_csv(.......) on my Kaggle kernel.
I am not quite sure what the issue.
Any thoughts of what can i do now to fix that?
Thanks.


texts: Regression with Neural Networks in Tensorflow problem

texts:
I have recently started learning Neural networks and Python. I am trying out linear regression for a dataset with 14 features and 1 outcome. I have divided the data into training and test data. I have experimented with many parameters (learning rate, nodes per layer, number of layers, number of steps and optimization algorithm) but my test errors are as high as 150%.

I have posted my code below along with the cost curve (cost vs epochs).
Where am making a mistake and what should I change? Or can you suggest some other important checks?

import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf

# importing features and observations data for training and validation
training_filename_X = "training_set_X.csv"
training_filename_Y = "training_set_Y.csv"
test_filename_X = "test_set_X.csv"
test_filename_Y = "test_set_Y.csv"
training_features = np.loadtxt(training_filename_X, delimiter=',')
training_observations = np.loadtxt(training_filename_Y, delimiter=',')
test_features = np.loadtxt(test_filename_X, delimiter=',')
test_observations = np.loadtxt(test_filename_Y, delimiter=',')

# normalizing training data
training_features_stddev_arr = np.std(training_features, axis=0)
training_features_mean_arr = np.mean(training_features, axis=0)
normalized_training_features = (training_features-training_features_mean_arr)/training_features_stddev_arr

# normalizing validation data with training set mean and standard deviation
normalized_validation_features = (validation_features-training_features_mean_arr)/training_features_stddev_arr

# normalizing test data with training set mean and standard deviation
normalized_test_features = (test_features-training_features_mean_arr)/training_features_stddev_arr

# layer parameters
n_nodes_hl1 = 20
n_nodes_hl2 = 20
n_nodes_hl3 = 20
no_features = 14
learning_rate = 0.01
epochs = 200

cost_history = np.empty(shape=[1], dtype=float)

X = tf.placeholder(tf.float32)
Y = tf.placeholder(tf.float32)

# defining weights for each layer taken from a normal distribution with variance 2/n
hl1_weight = tf.Variable(tf.random_normal([no_features, n_nodes_hl1], stddev=np.sqrt(2/no_features)))
hl2_weight = tf.Variable(tf.random_normal([n_nodes_hl1, n_nodes_hl2], stddev=np.sqrt(2/n_nodes_hl1)))
hl3_weight = tf.Variable(tf.random_normal([n_nodes_hl2, n_nodes_hl3], stddev=np.sqrt(2/n_nodes_hl2)))
output_weight = tf.Variable(tf.random_normal([n_nodes_hl3, 1], stddev=np.sqrt(2/n_nodes_hl3)))

# defining biases for each layer
hl1_bias = tf.Variable(tf.random_uniform([n_nodes_hl1], -1.0, 1.0))
hl2_bias = tf.Variable(tf.random_uniform([n_nodes_hl2], -1.0, 1.0))
hl3_bias = tf.Variable(tf.random_uniform([n_nodes_hl3], -1.0, 1.0))
output_bias = tf.Variable(tf.random_uniform([1], -1.0, 1.0))

# defining activation functions for each layer
hl1 = tf.sigmoid(tf.matmul(X, hl1_weight) + hl1_bias)
hl2 = tf.sigmoid(tf.matmul(hl1, hl2_weight) + hl2_bias)
hl3 = tf.sigmoid(tf.matmul(hl2, hl3_weight) + hl3_bias)
output = tf.matmul(hl3, output_weight) + output_bias

# using mean squared error cost function
cost  = tf.reduce_mean(tf.square(output - Y))

# using Gradient Descent algorithm
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)

init = tf.global_variables_initializer()

# running the network
with tf.Session() as sess:
    sess.run(init)

    for step in np.arange(epochs):
        sess.run(optimizer, feed_dict={X:normalized_training_features, Y:training_observations})
        print (sess.run(cost, feed_dict={X:normalized_training_features, Y:training_observations}))
        cost_history = np.append(cost_history, sess.run(cost,feed_dict={X:normalized_training_features, Y:training_observations}))

    pred_y = sess.run(output, feed_dict={X:normalized_test_features})
    print (sess.run(output, feed_dict={X:normalized_test_features}))
    mse = tf.reduce_mean(tf.square(pred_y - test_observations))
    print("MSE: %4f" % sess.run(mse))

# plotting the cost history
plt.plot(range(len(cost_history)), cost_history)
plt.axis([0, epochs, 0, np.max(cost_history)])
plt.show()


texts: Why is random forest an improvement of decision tree?

texts:
Let's assume that we have a binary classification problem, and we built a decision tree on our data set.

Assuming that we have 5 features, then the decision tree, in the first step, will choose the best feature of the 5, and on this feature it will choose the best threshold in order to split the data set, and then continue to make the tree deeper etc. The definition of best is the lowest classification error.

My question is: Since the decision tree, on each step, chooses the best feature to split on, and the best threshold to split on, then why random forest (which is many decision trees), is an improvement of decision trees? Shouldn't a decision tree be sufficient?

UPDATE

I more mean it like: If you have a decision tree classifier, and a random forest classifier with the same parameters, when possible, (max_depth, number of children etc), will the decision tree classifier score the same on the training set, with the random forest classifier?


texts: Solving multi label image classification using TimeDistributed dense layer

texts:
I have a multi label image dataset having 5 labels. Each image can have more than one label at the same time. I am using a convolutional neural network to extract features and those extracted features I am giving to RepeatVector layer to create 5 copies of extracted features and after RepeatVector layer I have connected TimeDistributed layer with TimeDistributed(Dense(2)).

y_train is a 3D array and its shape is (1600, 5, 2) and x_train is an array of images.

For example:

>>> x_train.shape
(1600, 3, 100, 100)

>>> y_train.shape
(1600,5,2)

>>> y_train[0] =
array([[0, 1],   # [0,1] = 1 label present and  [1,0] = 0 label abset
       [1, 0],
       [1, 0],
       [1, 0],
       [1, 0]])

Code:

def get_label(y):
  tmp = []
  d = {0:[1,0],1:[0,1]}   # 0 absent 1 present
  for i,value in enumerate(y):
    tmp.append( d[value] )
  return tmp

X,Y= get_data()
Y = Y.tolist()
y = []
for value in Y:
  y.append(get_label(value))

Y = np.array(y,dtype=int)

x_train, x_test, y_train, y_test = train_test_split(X,Y,test_size =0.2,random_state=100)

img_channels = 3
img_rows     = 100
img_cols     = 100
nb_classes   = 5

model = Sequential()

model.add(Convolution2D(32, 3, 3, border_mode='same',input_shape=(img_channels, img_rows, img_cols)))
model.add(Activation('relu'))
model.add(Convolution2D(32, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

model.add(Convolution2D(64, 3, 3, border_mode='same'))
model.add(Activation('relu'))
model.add(Convolution2D(64, 3, 3))
model.add(Activation('relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.25))

model.add(Flatten())
model.add(Dense(512))
model.add(Activation('relu'))
model.add(RepeatVector(nb_classes))
model.add(TimeDistributed(Dense(2)))
model.add(Activation('softmax'))

# let's train the model using SGD + momentum (how original).
# opt = RMSprop(lr=0.001, rho=0.9, epsilon=1e-06)
opt = SGD(lr=0.01, momentum=0.0, decay=1e-6, nesterov=False)
model.compile(loss='categorical_crossentropy', optimizer=opt,metrics=['accuracy'])
model.fit(x_train,y_train,nb_epoch=10,batch_size=32,validation_data=(x_test,y_test),shuffle=True)
out = model.predict_classes(x_test)

But after training, I get all zeros for the test set. Is this approach wrong?


texts:
So I was applying the random forest classifier model to a problem,however it showed this error ,even though the columns in X and Y of my dataset are equal. How can I resolve this?

ValueError                                Traceback (most recent call last)
<ipython-input-123-8b3f6408c588> in <module>
      1 from sklearn.metrics import confusion_matrix,accuracy_score
----> 2 cm = confusion_matrix(y_pred4,Y_test)

~\Anaconda3\lib\site-packages\sklearn\metrics\classification.py in confusion_matrix(y_true, y_pred, labels, sample_weight)
    251
    252     """
--> 253     y_type, y_true, y_pred = _check_targets(y_true, y_pred)
    254     if y_type not in ("binary", "multiclass"):
    255         raise ValueError("%s is not supported" % y_type)

~\Anaconda3\lib\site-packages\sklearn\metrics\classification.py in _check_targets(y_true, y_pred)
     69     y_pred : array or indicator matrix
     70     """
---> 71     check_consistent_length(y_true, y_pred)
     72     type_true = type_of_target(y_true)
     73     type_pred = type_of_target(y_pred)

~\Anaconda3\lib\site-packages\sklearn\utils\validation.py in check_consistent_length(*arrays)
    203     if len(uniques) > 1:
    204         raise ValueError("Found input variables with inconsistent numbers of"
--> 205                          " samples: %r" % [int(l) for l in lengths])
    206
    207

ValueError: Found input variables with inconsistent numbers of samples: [7500, 2500]


texts: Communication between name node, data node and client in hadoop by analysing Packet Capturing

texts:
if I say that in hadoop cluster system...       A client has 10GB data to send to name node. After analysing the "packet capturing" actually the name node doesn't divide the data for data node, actually all the communication happens only between client and data node , after taking the IPs of data node from name node, client directly connect to data node and after that he decide , in how many partitions  he is sending the data by dividing the blocks.
And then the data is receives by the data node and during this time name node contact to the data nodes to be prepare of replicating these data.

So my question is that the by this process I realized that the data sent by client is receives by data node in serial manner and date node copies that data regularly and hand to hand. Then if I sending a file of 100GB and usually it takes 100 minutes but after using tha Hadoop my speed and time is consuming more than before because the data node receives the data serially from client .
"IS THIS A MAJOR DISADVANTAGE OF HADOOP??"


texts:
I have hourly power consumption data for 30 days. On representing, each day data using a separate line, I get a plot as

I want to highlight the days with abnormally high consumption (in other words, the outlier days). I think that the current plot is too much congested. Is there any other better representation to show the outlier days?