texts: Tips for checking data integrity / data sanity?

texts:
I've read a few vague articles and watched a couple of YouTube videos on data integrity and data sanity, but none of them have mentioned ways to actually check these on datasets.

I am interested in knowing the steps to check such factors on datasets.

Kindly let me know if this is too broad, I can think of more specific questions.


texts: Isolation Forest height limit absent in SkLearn implementation

texts: In the original publication of the Isolation Forest algorithm, the authors mention a height limit parameter to control the granularity of the algorithm. I did not find that explicit parameter on the Sklearn implementation of the algorithm, and I was wondering whether it is possible to control granularity in some other way?


texts: Classifier and Technique to use for large number of categories

texts:
I am designing a scikit learn classifier for a sequence labelling task which has 5000+ categories and training data is at least 80 million and may grow upto an additional 100 million each year. I have already tried with all the categories but it generates classifiers in the order of 100s of GBs binary file. So I think that having one classifier for each category would be helpful and would also help me to fine tune features for each category thereby improving accuracy, but this means 5k+ classifiers for each of these categories. So how to handle this large data requirements and which incremental classifiers to use for this case , considering the fact that I will keep on getting additional training data as well as may discover new categories?

The number of features are about 100+ and due to the sequence labelling task contiguous sequence of training samples share same features values. The feature values are mostly text based and most are categorical with long text based values  with large cardinality i.e many features may have huge number of possible values.

The available RAM IS 32gb with 8 core CPU. On a small scale I tried Multinomial NB and linear SGD with sparse matrices which are extremely sparse. Used the scikit learns Dictvectorizer to vectorize the feature dictionary. Also will pandas dataframes help to optimize the overall configuration?

Due to the scale of data involved I have only used about 20% of the data. And the features itself are 56GB of data from the 500MB input data. For the restricted training data, The precision too is very low does not reach above 12% with a very low  recall too(1-2%). Have gone through vowpal wabbit and  found the SEARN task but it seems that it is now no longer available in the latest version .


texts: Cluster algorithm to group events in more general domains

texts:
I've a list of 1,300 news events, represented by only three terms coming from running LDA topic model on thousands of tweets. Here's some of them as an example:

['manchester,bony,city', 'attack,claims,responsibility', 'police,officers,nypd',
 'goal,arsenal,liverpool', 'test,pakistan,sunday', 'obama,ukraine,merkel', ...]

I need to group them in more general domains (Politics, Sport, Health, Economy, etc.).

Which kind of clustering algorithms could I use (in Python)?

Or maybe, can I use LDA topic model, even if I don't have documents but only three words?


texts: Hyperparameter optimization, ensembling instead of selecting with CV criteria

texts: While burning CPUs performing a CV selection on a thin grid put on some hyperparameter space. I am using the `scikit-learn' API, for which the end result is a single point on the hyperparameter space, for which the performance is the best according to chosen metric. It looks like a lot of information is trashed, we are probing the performance landscape of the hyperparameters and keeping a single point estimate. Information such as having many modes, local maximas, maybe far apart. An ensembler method could make good use of such information. Is there a theory built over ensembling a diversity of hyperparameter estimates?


texts: Changing the default scale of a plot to a "custom scale"

texts:
I am trying to change the scale of the x_axis for a plot.

The default is generating divisions of 20 units (0-20-40-60-80-100-120).

I tried changing to log, but then I get 0-10-100, which helps understand the data, but I would like to try to see divisions with 10 units because my data is not logarithmic (its linear, but there are many values in the 0-10 range, which results in a plot that is very hard to read).

How can I achieve that effect, or another one that will help me visually the data?

The code is below:

import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline
sns.set()
plot = data3.plot(kind="scatter", x="A", y="B") # data3 is the name of my data frame

plt.xscale('log') # this was my change (plt is correct)

plot

I also tried the following (instead of the 'log' line above), but it gives the default (0-20-40-etc):

plt.xticks = [0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100]


texts:
I have noticed that more often when training huge Deep Learning models on consumer GPUs (like GTX 1050ti) The network often doesn't work.

The reason is that the GPU just doesn't have enough memory to train the said network. This problem has solutions like using the CPU's cache memory for storage of things that are not being actively being used by GPU.

  So my question is - Is there any way to train models on CUDA with memory drawn from System RAM? I understand that the tradeoff would involve speed and accuracy but it JUST might work to train a BERT model on my 1050ti


texts:
I am very confused between one hot encoding and word embedding in terms of structure of the network and how it reduces the dimensionality.

I am currently using encog with c# which has some documentation but not many differing examples.

If I take an example of text classification by neural networks and have:
1000 different texts
10,000 unique words

What I was previously doing was taking each text and then working out the unique words and putting them in a list. I would then work out the number of times each unique word appeared in a text and I would end up with a matrix of:

1000 x 10,000 i.e. for each text a count of the number of times each unique word appears (text as rows and unique words (features) as columns).

I would then normalise this and use equilateral encoding (similar to one hot).

My network would then consist of 10,000 input neurons (one for each word (feature)) and then a hidden layer and then an output layer trying to predict a classification of an output column with 5 different values.

This appears to be totally wrong but it did train and pretty accurately. I'm trying to work out with one hot encoding what exactly that would mean in terms of matrix sizes, what those matrices consist of and the layout of the network.

This is also related to using word embedding as when I use this I get the following size: 1000 (number of texts), 872 (text size), 300 (vector size). I therefore have 3 dimensions so that made me think in one hot encoding I should probably have: 1000, 872, 10000 (unique words).

If someone could explain where I am going wrong here and the difference between how I would approach one hot encoded text classification vs an embedded layer using:

1000 texts, 10000 unique words, 872 text length and 300 vector size


texts: How to prepare the data for text generation task

texts:
First, I'm not sure whether the model contains the encoder during training.

EOS means end-of-sentence. Encoder and decoder are part of transformer network.

If without-encoder, training time:

target: [E, F, G, H, EOS]
decoder input: [0, E, F, G, H]

If without-encoder, testing time:

decoder input: [0]

If with encoder, training time:

encoder input: [A, B, C, B]
target: [E, F, G, H, EOS]
decoder input: [0, E, F, G, H]

If with-encoder, testing time:

encoder input: [A, B, C, D]
decoder input: [0]

Am I exact right?


texts: How do I calculate the range of a F1-score from a confusion matrix of 3 class,A,B,C

texts: Is there any support function to calculate the average F1-score range?


texts: MAE estimation for k-fold cross-validation

texts:
I have code that estimates RMSE for k-fold cross-validation and I think it is correct (from book: Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition by Aurélien Géron)

scores = cross_val_score(forest_reg, a, b, scoring="neg_mean_squared_error", cv=10)
print(pd.Series(np.sqrt(-scores)).describe())

So what about MAE? Should I use (with sqrt):

scores = cross_val_score(forest_reg, a, b, scoring="neg_mean_absolute_error", cv=10)
print(pd.Series(np.sqrt(-scores)).describe())

or this (without sqrt):

scores = cross_val_score(forest_reg, a, b, scoring="neg_mean_absolute_error", cv=10)
print(pd.Series(-scores).describe())

Also for MAE estimation, it should be  -scores or scores?


texts: Hierarchical clustering with the consensus matrix as similarity matrix

texts:
I'm following this article on consensus clustering in Python programming. On page 7 the authors state that "The consensus matrix lends itself naturally to be used as a visualization tool to help assess the clusters’ composition andnumber. In particular, if we associate a color gradient to the 0–1 range of real numbers, so that white corresponds to 0, and dark red corresponds to 1, and if we assume the matrix is arranged so that items belonging to the same cluster are adjacent to each other (with the same item order used to index both the rows and the columns of the matrix), a matrix corresponding to perfect consensus will be displayed as a color-coded heat map characterized by red blocks along the diagonal, on a white background.", see the image below.

The consensus matrix itself is a (N × N) matrix that stores, for each pair of items, the proportion of clustering runs in which two items are clustered together. The consensus matrix is obtained by taking the average over the connectivity matrices of every perturbed dataset. To go from a consensus matrix to the visualization above, the authors say the following: "We can use the consensus matrix itself to determine the optimal item order. In particular, if we carry out hierarchical clustering with the consensus matrix as similarity matrix, the induced dendogram will have its leaves arranged so as to have items with highest consensus index adjacent to each other, thus maximizing the block-diagonal nature of the heat map ordered accordingly (it is important to emphasize again that the same item order is used to index both the rows and the columns of the matrix)."
I have my consensus matrix after 10 times sampling and clustering my dataset with 500 items. The consensus matrix looks like:

Where a value of 1 or 0 indicates perfect agreement/consensus on the clustering results of the 10 times sampling. However, I fail to understand how we end up with the visualization above. I have a few questions:

The authors state that we use the consensus matrix as a similarity matrix, but doesn't hierarchical clustering need a distance matrix? For example in the Python sklearn implementation of AgglomerativeClustering it says: "If “precomputed”, a distance matrix (instead of a similarity matrix) is needed as input for the fit method."
What do the authors mean with "it is important to emphasize again that the same item order is used to index both the rows and the columns of the matrix"?


texts: Is it possible to predict groups vs groups in machine learning?

texts:
I have a problem that I think machine learning can solve. I'm new to it however and don't know where to start so any pointers to appropriate solving methods/guides etc for the problem are appreciated.
I've searched a bit and couldn't find any questions similar to this, but most likely because I dont know exactly what to search for:

Lets say we have a bunch of fruits: a,b,c,d,e,f which all have two factors associated with it, a price and a quality. Every day in a store the customers can choose between two bundles(g1,g2), each consisting of two fruits with varying price and quality(p,q). The 4 fruits every day must be unique, so g1 cannot have a fruit in g2.
The customers then buy one bundle each. They value the price and quality, but are probably also influenced by outside factors such as what fruit it is and what it is combined with.

Is it possible to create a training set from this? I came up with two ways to represent this:
The simplest way i could think of: let a fruit x be represented as x:{x_p,x_q}, a group as g1(a,b), and a training element would then be defined as day(g1,g2) which we will use to train our machine.
The second way was to let the fruits contain a present flag to mark if they are in a group or not, x:{x_p,x_q,present} and a group would then become g1(a,b,c,d,e,f) and a day still day(g1,g2)

Is it possible to do it the first way without data for several fruits, or is it just implementation specific?
How would one train for the mirroring case of day(g1(a,b),g2(c,d)) vs day(g1(c,d),g2(a,b)), by just creating the mirrored training element or is something that the machine could figure out on its own?
While this is a minimum example, what would be different if we were to cover more fruits(5/bundle) from a larger fruit pool(20)?
Can we use it to predict which bundle is more likely to be picked even if that group/fruit combo hasn't been evaluated in the set?

Another way I thought of was training on the groups, but I think that would result in a flat score for the groups and most likely miss relationships between the groups, such as g1 is preferred over g2 which is preferred over g3 which is preferred over g1? Is this true or would a machine be able to understand that too?

I was looking a bit at the javascript libraries brain and mind and thought that it might be possible with one of them, but I dont know what type of algorithm/solving method I'm looking for. Any suggestions for libraries to solve this in, preferrably python/php/java/javascript are appreciated!

Also if someone could add suitable tags to the question would be great!

EDIT: same as with the group mirroring, how do we distinguish the groups, do we need to sort the fruits alphabetically or can a machine handle it too? (apple, banana) should be identical to (banana,apple)


texts:
I need to use an Excel file that have many merged cells with no specific order.

I guess that the file creator thought that it would be more readable to merge the cells that have the same value.

However I would like to get a clean source file with one cell per row and column number in order to work further with it in other tools such as Jupyter Notebooks.

If I undo the cells merge for all, I get many empty cells.

Let's say that I get the following type of merged cells :

-----------------------
1 |
2 |  "Content"
3 |
-----------------------

If I cancel cells merging for all the datasheet I get :

-----------------------
1 |  "Content"
-----------------------
2 |
-----------------------
3 |
-----------------------

What I need is :

-----------------------
1 |  "Content"
-----------------------
2 |  "Content"
-----------------------
3 |  "Content"
-----------------------

This can be cleaned by hand cell by cell but the dataset is huge. Is there a built-in method or a macro to do this automatically on all cells, or some other tool available to do this?


texts:
I want to draw plot by python based on this data

Id     Duration (s)       BinaryLabel
1          5                 correct
2          4                incorrect
3          2                incorrect
4          3                incorrect
5          9                 correct
6          6                 correct
7          12               incorrect

My data is with 800 rows. and the duration is going from 0 to 55 seconds

My X axis contain duration column and I want it to be divided like: 0-----5-----10------15------20-----...-----55

and the Y should contain the percent of 'correct' values.

I.e. :

from 0 to 5, I will have 25% correct,

from 5 to 10 I will have 100% correct .etc.


texts: Does reinforcement learning only work on grid world?

texts:
Does reinforcement learning always need a grid world problem to be applied to?

Can anyone give me any other example of how reinforcement learning can be applied to something which does not have a grid world scenario?


texts: Can I use unsupervised learning followed by supervised learning?

texts:
I have a question about classifying documents using supervised learning and unsupervised learning.

For example: - I have a bunch of documents talking about football. As we know, football has a different meaning in UK, USA, and Australia. Therefore, it is difficult to classify these documents to three different categorizations (which are soccer, American football and Australian football).

My approach tries to use cosine similarity terms based on unsupervised learning. After we use the cluster learning, we are able to create a number of clusters based on cosine similarity, where each cluster will contain similar documents terms. After we create the clusters, we can use a semantic feature to identify these clusters depending on a supervised model like SVM to make accurate categorizations.

My goal is to create more accurate categorizations, because if I would like to test a new document, I would like know if this document can be related to these categorizations or not.


texts: multipying negated gradients by actions for the loss in actor nn of DDPG

texts:
In this Udacity project code that I have been combing through line by line to understand the implementation, I have stumbled on a part in class Actor where this appears on line 55 here:  https://github.com/nyck33/autonomous_quadcopter/blob/master/actorSolution.py

# Define loss function using action value (Q value) gradients
        action_gradients = layers.Input(shape=(self.action_size,))
        loss = K.mean(-action_gradients * actions)

The above snippet seems to be creating an Input layer for action gradients to calculate loss for the Adam optimizer (in the following snippet) that gets used in the optimizer but where and how does anything get passed to this action_gradients layer? (https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Adam)  get_updates function for optimizer where the above loss is used:

# Define optimizer and training function
optimizer = optimizers.Adam(lr=self.lr)
updates_op = optimizer.get_updates(params=self.model.trainable_weights, loss=loss)

Next we get to train_n a K.function type function (also in class Actor):

self.train_fn = K.function(
            inputs=[self.model.input, action_gradients, K.learning_phase()],
            outputs=[],
            updates=updates_op)

of which action_gradients are the (gradient of the Q-value w.r.t. actions from the local critic network, not target critic network).

The following are the arguments when train_fn is called:

    """
    inputs = [states, action_gradients from critic, K.learning_phase = 1 (training mode)],
    (note: test mode = 0)
    outputs = [] are blank because the gradients are given to us from critic and don't need to be calculated using a loss function for predicted and target actions.
    update_op = Adam optimizer(lr=0.001) using the action gradients from critic to update actor_local weights
"""

So now I began to think that the formula for deterministic policy gradient here:  https://pemami4911.github.io/blog/2016/08/21/ddpg-rl.html
would be realized once the gradient of the Q-values w.r.t. actions were passed from the critic to the actor.

I think that the input layer for action_gradients mentioned at the top is trying to find the gradient of the output of actor w.r.t. to parameters so that it can do the multiplication pictured in the photo. However, to reiterate, how is anything passed to this layer and why is the loss calculated this way?

Edit:  I missed a comment on line 55

# Define loss function using action value (Q value) gradients

So now I know that the action_gradients input layer receive the action_gradients from the critic.
Apparently it is a trick used by some implementations such as Openai Baselines:  https://stats.stackexchange.com/questions/258472/computing-the-actor-gradient-update-in-the-deep-deterministic-policy-gradient-d

But still, why is the loss calculated as -action_gradients * actions ?


texts: On assiging weights for unbalanced classes

texts: Consider a dataset that will be split into train and test. The model will be learned using the train set and evaluated using the unseen test set. Now the dataset is unbalanced -- it contains more examples belonging to a particular kind of class. In that case, one of the ways to balance it (apart form the ones mentioned here: Train, test split of unbalanced dataset classification) is by assigning weights based on the samples. What is the proper way to assign weights? Should I assign weights on the full dataset and then split into train and test?


texts: Case when Out of Bag Error and Test error differs a lot in Random Forest

texts:
I'm using random forest and the out of bag error for the level of one class is very different to its test error. I'm working with a cutt-of equal to c(0.2,0.8). Here's the case:

fmla <- as.formula(paste("ex ~ ", paste(names(muestra.fullarbol[,-c(1,2,3,9,10,11,12,17,19,20,21,22,23,24,29,31,32,33,34,35,36,47)]), collapse= "+")))
> bosque <- randomForest(fmla , data=muestra.fullarbol ,mtry=12, ntree=1000  , cutoff=c(0.2,0.8),importance=TRUE)
> bosque

Call:
 randomForest(formula = fmla, data = muestra.fullarbol, mtry = 12,      ntree = 1000, cutoff = c(0.2, 0.8), importance = TRUE)
               Type of random forest: classification
                     Number of trees: 1000
No. of variables tried at each split: 12

        OOB estimate of  error rate: 15.81%
Confusion matrix:
     No Si class.error
No 3999  1     0.00025
Si  758 42     0.94750

As we see, the out of bag error for the level "SI" is 0.94750. However If I use the test set to get a sense of the error, the result is very different:

res.arbol <- predict(bosque,test.fullarbol,type="class")
> summary(res.arbol)
  No   Si
7761   43
> table(res.arbol,test.fullarbol$ex)

res.arbol   No   Si
       No 6937  824
       Si    4   39
> prop.table(table(res.arbol,test.fullarbol$ex),1)

res.arbol         No         Si
       No 0.89382811 0.10617189
       Si 0.09302326 0.90697674

Now we see thae error rate for the test set in the class "Si" is very low ,  it's equal to 0.093 and It doesn´t make sense to me.

I guess that the cut off is working just to predict out of the sample(muestra.fullarbol), but I'm not sure. What can be the reason of that huge difference?


texts: how I get different array size in calculating tpr?


texts: Assigning values to missing target vector values in scikit-learn

texts:
I have a dataset containing data on temperature, precipitation, and soybean yields for a farm for 10 years (2005 - 2014). I would like to predict yields for 2015 based on this data.

Please note that the dataset has DAILY values for temperature and precipitation, but only 1 value per year for the yield (since harvesting of crop happens at end of growing season of crop).

I would like to build a regression or some other machine learning based model to predict 2015 yields, based on a regression/some other model derived by studying the relation between yields and temperature and precipitation in previous years.

As per, Building a machine learning model to predict crop yields based on environmental data, I am using sklearn.cross_validation.LabelKFold to assign each year the same label.

The question is that since I have a single target value per year, do I need to interpolate to fill in target values for all the other days of the year? Should I just use the same target value for each day of the year?


texts: Understanding action space in stable baselines

texts:
I was trying to write reinforcement learning agent using stable-baselines3 library. The agent(abservations) method should return action. I went through different models API (like PPO) and they do not really allow us to specify action space. Instead action space is specified in environment.
This notebook says:

The type of action to use (discrete/continuous) will be automatically deduced from the environment action space.

So, it seems that "model" deduce action space from environment.
Q1. But exactly how?
Q2. Also how my agent(observations) method should return action? By returning action returned by model.predict()?


texts: Genetic algorithms(GAs): to be considered only as optimization algorithms? Are GAs used in machine learning any way?

texts:
As a quick question, what are genetic algorithms meant to be used for? I read somewhere else that they should be used as optimization algorithms (similar to the way we use gradient descent to optimize the best parameters search with a linear regression, neural network...).
If so, why are these GAs not so much present in machine learning (or at least I did not see it too much in the literature)?


texts: Force Matching in Coarse Grained Molecular Dynamics with Jax - Forces do not match when neglecting energy loss

texts:
I am currently exploring force matching approaches for molecular dynamic simulations.
As I am still in an exploration state, I'd tried investigated
Force Matching Neural Network Colab Notebook
corresponding to
Unveiling the predictive power of static structure in glassy systems.
They train a graph neural network to match to estimate forces from positions.
Therefore, they calculate a loss where they match energy and forces.
Loss = $(energy_{predicted} - energy_{target})^2 + ( Forces_{predicted} - Forces_{target})^2$
where the Energy is defined as $U(x,\phi)$ and the force is defined as $-\frac{dU}{dx} = F$.
When neglecting the energy loss and purely matching forces, the prediction seems to converge to 0. This is not intuitive, as Coarse Graining Molecular Dynamics with Graph Neural Networks seem to train their GNN just with Force matching. The question is: Does anybody knows why the neural networks behave like this.
To reproduce my observations:
Change the loss-function in 1 to:
@jit
def loss(params, R, targets):
  return force_loss(params, R, targets[1])

and the training to:
train_epochs = 20

opt_state = opt.init(params)

train_energy_error = []
test_energy_error = []

for iteration in range(train_epochs):
  train_energy_error += [float(np.sqrt(force_loss(params, batch_Rs[0], batch_Fs[0])))]
  test_energy_error += [float(np.sqrt(force_loss(params, test_positions, test_forces)))]

  draw_training(params)

  params, opt_state = update_epoch((params, opt_state),
                                   (batch_Rs, (batch_Es, batch_Fs)))

  onp.random.shuffle(lookup)
  batch_Rs, batch_Es, batch_Fs = make_batches(lookup)

as well as the visualization of the trainging:
 def draw_training(params):
  display.clear_output(wait=True)
  display.display(plt.gcf())
  plt.subplot(1, 2, 1)
  plt.semilogy(train_energy_error)
  plt.semilogy(test_energy_error)
  plt.xlim([0, train_epochs])
  format_plot('Epoch', '$L$')
  plt.subplot(1, 2, 2)
  predicted = vmap(force_fn, (None, 0))(params, example_positions).reshape((-1,))
  plt.plot(example_forces.reshape((-1,)), predicted, 'o')
  #plt.plot(np.linspace(-400, -300, 10), np.linspace(-400, -300, 10), '--')
  format_plot('$E_{label}$', '$E_{prediction}$')
  finalize_plot((2, 1))
  plt.show()

to reproduce my observations.


texts: 'list' object has no attribute 'values' when we are using append in python

texts:
Here I have a dataset with three inputs. Here I generated y value using append. After the append I got the output like this:

 y.append(rec.iloc[0]['y'])

Then I tried to develop neural network model with these values. Before that I tried to scale the y value.
my code:

y =y.values().astype(int)
scaler_y = preprocessing.MinMaxScaler(feature_range =(0, 1))
y = np.array(y).reshape([-1, 1])
y = scaler_y.fit_transform(y)

Then I got an error :

AttributeError                            Traceback (most recent call last)
<ipython-input-254-2ec9d2fcbffd> in <module>()
----> 1 y = y.values().astype(int)

 AttributeError: 'list' object has no attribute 'values'

Can anyone help me to solve this error?


texts: Use spark_csv inside Jupyter and using Python

texts:
My ultimate goal is to use Jupyter together with Python for data analysis using Spark. The current hurdle I face is loading the external spark_csv library. I am using Mac OS and Anaconda as the Python distribution.

In particular, the following:

from pyspark import SparkContext
sc = SparkContext('local', 'pyspark')
sqlContext = SQLContext(sc)
df = sqlContext.read.format('com.databricks.spark.csv').options(header='true').load('file.csv')
df.show()

when invoked from Jupyter yields:

Py4JJavaError: An error occurred while calling o22.load.
: java.lang.ClassNotFoundException: Failed to find data source: com.databricks.spark.csv. Please find packages at http://spark-packages.org

Here are more details:

Setting Spark together with Jupyter

I managed to set up Spark/PySpark in Jupyter/IPython (using Python 3.x).

System initial setting

On my OS X I installed Python using Anaconda. The default version of Python I have currently installed is 3.4.4 (Anaconda 2.4.0). Note, that I also have installed also 2.x version of Python using conda create -n python2 python=2.7.

Installing Spark

This is actually the simplest step; download the latest binaries into ~/Applications or some other directory of your choice. Next, untar the archive tar -xzf spark-X.Y.Z-bin-hadoopX.Y.tgz.
For easy access to Spark create a symbolic link to the Spark:

ln -s ~/Applications/spark-X.Y.Z-bin-hadoopX.Y ~/Applications/spark

Lastly, add the Spark symbolic link to the PATH:

export SPARK_HOME=~/Applications/spark
export PATH=$SPARK_HOME/bin:$PATH

You can now run Spark/PySpark locally: simply invoke spark-shell or pyspark.

Setting Jupyter

In order to use Spark from within a Jupyter notebook, prepand the following to PYTHONPATH:

export PYTHONPATH=$SPARKHOME/python/lib/py4j-0.8.2.1-src.zip:$SPARKHOME/python/:$PYTHONPATH

Further details can be found here.


texts: Visualization with many lines, colors, and markers

texts:
I have a bunch of plots as the one reported below. The data is from measurements performed on different times and different days.
In the plot (which is a cumulative distribution function, if that matters), the colors differentiate data relevant to different days; the markers are used to further differentiate the data within each day.
The problem is that the plot is very crowded and a bit ugly. Some markers can be barely seen.
Question: Any idea how I can better plot this data?
As you can see I tried to scatter the markers so that they would not appear in the same position.
The code I used to create the markers (with python/matplotlib):
marker_list = ['$A$', '$B$', '$C$', '$D$', '$E$', '$F$', '$G$', '$H$', '$I$', '$J$',
               '$K$', '$L$', '$M$', '$N$', '$O$', '$P$', '$Q$', '$R$', '$S$', '$T$',
               '$U$', '$V$', '$W$', '$X$', '$Y$', '$Z$', r'$\Gamma$', r'$\Lambda$',
               r'$\Omega$', r'$\Psi$',]

'set marker position'
rnd_lst = [np.random.randint(-10, 10) for i, _ in enumerate(marker_list)]
marker_pos = [[25+x, 45+x, 65+x, 87+x] for x in rnd_lst]    # fixed points + random number to scatter the markers

cc = plt.cycler(marker=marker_list, markevery=marker_pos)
plt.rc('axes', prop_cycle=cc)
plt.rc('lines', markersize=18)

However, it would be nice to "distribute the markers in the y direction". For example, it'd be nice to show the markers only where the CDF is above 0.25.
Question: how to limit the markers to a specific section of the y axis?
EDIT
I guess my best option is to use linestyles instead markers. However, using more than 4 or 5 linestyles is also bad because they will be barely distinguishable from one another. To solve I can use a combination of linestyles and markers when I have more than say 4-5 lines with same color.  The problem with this solution is that I find it not immediate to automate,  at least not by using the cycler as shown above. Any help on this would be also appreciated :)


texts: normalize dependent variable with respect to one independent variable

texts:
I have a data set with 50-60 independent variables and 1 categorical (1-0) dependent variable. Out of all the independent variables say, A is one variable which has majority of the effect on the target variable. For ex

A        target variable
a              1
a              1
a              1
a              1
b              0
b              1
b              1
b              1
c              0
c              0
c              0
c              1
c              1

Clearly whatever analysis I do here, whatever probabilities I calculate for With respect to other variables, I would not be able to get the actual picture with respect to other variables.

Also I do not want to build any model at this point, I just want to see the individual effects of other variables on the dependent variable after removing the effect of variable 'A' i.e. normalizing dependent variable with respect to 'A'. How do I do? If the dependent variable was a continuous variable, I would have simply divided it by the respective averages and removed the effect. How do I do this in this case where the dependent variable is 1-0?


texts:
I developed an Android app that lets anyone upload pictures of encyclopedic things (bridges, museums, dishes, landscapes, paintings, etc) to Wikimedia Commons.

Unfortunately, 5% of the users find it funny to upload their own selfie. So I want to programmatically check whether the picture is a selfie or not, and if it probably is, warn them that selfies are off-topic.

As a data set, I have:

1000 pictures that I consider as undesirable selfies. It is in part subjective, but usually such pictures show one or two human faces taken from an arm's distance and random backgrounds.
1000 pictures that are not selfies (bridges, museums, dishes, etc, anything really). Tricky: this also includes pictures of famous people, usually they are easy to distinguish from a selfie because the persons are at a further distance. If you see an extended arm then you can be sure it is a selfie.

All pictures are taken with smartphones (hundreds of different models), they are JPG files of 2MB to 5MB in various sizes and ratios, in portrait or landscape mode.

I must use only open source, and the resulting detection code must run in less than a second on low-end Android phones.

What approach and steps does this task call for?


texts: Is there an appropriate sequence of parameters to be considered when building a pipeline in Scikit learn?

texts:
DISCLAIMER: I understand this could be considered a subjective type of question, but I'm hoping it is specific enough to not be flagged. Please give it a chance.

I am doing a GridSearch in scikit-learn on a corpus of 50 books (all kinda average commercially published texts...each ~200 pages of text).

As I began playing with GridSearch, I got to thinking. Surely there must be some sort of 'best practices' for building a pipeline?

Sure, I can include and remove parameters, but I'm really just sorta stabbing around in the dark for the best score.

Possibly, it could make sense to create a smaller, exploratory pipeline using the larger, more effective parameters first. After running this basic pipeline, more parameters that have a smaller effect (in general) could be run next.

For example, say I'm GridSearching a TfidfTransformer.  I am assuming that flipping the use_idf switch on or off is going to make a real impact on the scoring, whereas perhaps specifying alpha=0.000010 versus alpha=0.000015 won't.

In general, when using a pipeline, is there an alternative to "throw in the kitchen sink" that results in a more cognizant, informative (and possibly faster) experience?


texts: How do I use multilevel regression models?

texts:
I have crime event data rows:

dayofweek1, region1, hour1, crimetype1
dayofweek2, region2, hour2, crimetype2 ...

and I want to use them as factors to model crime rates/probabilities at the region level.

I also want to use the resulting  model to be able to input factor values to produce crime probabilities. e.g. on Sunday, in region1 there is a .03 chance of burglary at 3pm.

I think I should use a multilevel model, but everything I have found assumes a y value at the individual data row level which I do not have. All the row data are crimes.

Does anyone have an example of such a model (obviously not necessarily crime data, and preferably using python)?

Can the prediction bit be done?


texts: pytorch convolution with 0-stride along one dimension

texts:
For some square images, I'd like to use torch.nn.Conv2d with the kernel as a vertical block. As in, the kernel size is defined as max value of the first dimension by 1. Since the first dimension has no more room, I'd like to have 0 stride along that first dimension. I tried using the following definition:
self.conv1 = torch.nn.Conv2d(3, 32, (max_dim_0, 1),
                                    stride=(0, 1),
                                    padding=0,
                                    dilation=1,
                                    groups=1,
                                    bias=True,
                                    padding_mode='zeros')

However, this causes my training loop throws a RuntimeError: non-positive stride is not supported exception. Is there a way to define the kernel as a vertical block?


texts: Interpretation of variable or feature importance in Random Forest

texts:
I'm currently using Random Forest to train some models and interpret the obtained results.

One of the features I want to analyze further, is variable importance. The thing is I am not familiar on how to do a proper analysis of the results I got.
Let's say I have this table:

| Predictor  | Importance |
----------------------------
|   var_1    |   num_1    |
...
|   var_n    |   num_n    |

What is a proper analysis that can be conducted on the values obtained from the table, in addition to saying which variable is more important than another?

I was suggested something like variable ranking or using cumulative density function, but I am not sure how to begin with that.


texts: How to use data generator for regression keras?

texts:
I am using the Keras data generator to load data from a directory. I am basically dealing with a regression i.e there is a numerical value for each of my images in the range [-1 1]. I am using the following code to load images from the directory:
df=pd.DataFrame(columns=['Image','Steer'])
df['Image']=filenames
df['Steer']=steer

from sklearn.model_selection import train_test_split
train_x,test_x,train_y,test_y=train_test_split(df['Image'], df['Steer'], test_size=0.2, random_state=1)

train_df=pd.DataFrame(columns=['id','steer'])
train_df['id']=train_x
train_df['steer']=train_y

test_df=pd.DataFrame(columns=['id','steer'])
test_df['id']=test_x
test_df['steer']=test_y

images_dir = '/content/original/compressed/All_data/'
train_datagen = ImageDataGenerator(rescale = 1./255)
train_generator = train_datagen.flow_from_dataframe(dataframe=train_df, directory=images_dir,
                                              x_col="id", y_col="steer", has_ext=True,
                                              class_mode="other", target_size=(110, 220),
                                              batch_size=32)

test_datagen = ImageDataGenerator(rescale = 1./255)
test_generator = test_datagen.flow_from_dataframe(dataframe=test_df, directory=images_dir,
                                              x_col="id", y_col="steer", has_ext=True,
                                              class_mode="other", target_size=(110, 220),
                                              batch_size=32)
  import numpy as np
import pandas as pd

from skimage.io import imread
from skimage.transform import resize

import keras
from keras.models import Sequential, Model, load_model
from keras.layers import Input, Conv1D, Conv2D, MaxPooling1D, MaxPooling2D, Dense, Dropout, Activation, Flatten
from keras.layers.normalization import BatchNormalization
from keras.utils import to_categorical
from skimage.transform import resize
from tensorflow.keras import layers
import tensorflow as tf

  from keras.applications import VGG16

# include top should be False to remove the softmax layer
pretrained_model = VGG16(include_top=False, weights='imagenet')

base_model = VGG16(input_shape = (110, 220, 3), # Shape of our images
include_top = False, # Leave out the last fully connected layer
weights = 'imagenet')

for layer in base_model.layers:
    layer.trainable = False

from tensorflow.keras import layers
import tensorflow as tf

# Flatten the output layer to 1 dimension
x = layers.Flatten()(base_model.output)

# Add a fully connected layer with 512 hidden units and ReLU activation
x = layers.Dense(512, activation='relu')(x)

# Add a dropout rate of 0.5
x = layers.Dropout(0.5)(x)

# Add a final sigmoid layer for classification
x = layers.Dense(1, activation='linear')(x)

model = tf.keras.models.Model(base_model.input, x)
opt = keras.optimizers.Adam(learning_rate=0.001)
model.compile(loss = 'mean_absolute_percentage_error',optimizer=opt,metrics = ['mae'])

batch_size=128
history=model.fit(train_generator,
                   epochs = 15,
                   verbose = 1,
                   validation_data = test_generator
                  )

When I train the model using this code, a value of 0.221 is returned for the MAE for each epoch no matter what I do. I have changed the activation functions, optimizers, learning rate, etc but the result is the same every time. Am I loading in the data in a correct way?


texts: Sampling Big Data for Predictive Analytics in Python

texts:
In practice, how does one go about sampling a from big data set (eg. +/- 50 million distinct observations) to perform ML using Python?
Most non-parametric models (e.g., SVM, ensemble models) start to push computer resources with much smaller sets (e.g., 200 features, 200K observations).

How is this done in practice in industry?

Other questions here get at this but don't explicitly ask. So this is not a duplicate.
Thanks in advance.


texts: Cannot make user directory on a new CDH5 installation (Hadoop)

texts:
I download and installed CDH 5 package succesfully on a single linux node in pseudo-distributed Mode on my CentOS 6.5

Starting Hadoop and verifying it is Working Properly as in this link

I succesfully finished the following steps

Step 1: Format the NameNode.

Step 2: Start HDFS

Step 3: Create the /tmp Directory

Step 4: Create the MapReduce system directories:

Step 5: Verify the HDFS File Structure

Step 6: Start MapReduce

while following command in step 7 I get the following error.

Step 7: Create User Directories

$ sudo -u hdfs hadoop fs -mkdir -p /user/hadoopuser

mkdir: '/user/hadoopuser': No such file or directory

(where hadoopuser is my linux login username)

If I create the directory manually as /user/hadoopuser in the filesystem, it is not accepting.

How to success the step 7:?

Please provide the sloution to procced the remaining installation.


texts: Majority of feature maps of CNN are black

texts:
Assuming we have a following CNN : Conv->MaxPool->Conv->Maxpool->Linear.
What does it mean - intuitively - if the majority of the feature maps of the first convolutional layer are black i.e. have no activation?. Does that mean the corresponding filters which created the feature maps are useless i.e. extract no relevant information, so that we would achieve the same performance by removing them?


texts: Creating a new feature from an existing one using decision trees

texts:
Is it possible to create a new feature out of two, or more than two existing features using a decision tree?
If so, how, and can it produce features with good information value that can better help the model?


texts: measuring flip-flop behaviour across several topics

texts:
I'm trying to analyze a behavior called "sentiment flipping" of users in a dataset, but I'm not able to step on.

Let's suppose that I have two groups of users, say them good and bad users.

My dataset contains N tweets that classified into 6 topics. The tweets were created by the bad and good users.

The 6 topics are about general issues, but 3 of these topics are about organization/individuals supported (A) by the "bad" users and the other 3 are against (B) their ideologies.

The difference between the bad and good users in their tweeting behavior is:

The good user posted tweets in some of the topics (and maybe all of
them) without forcing "positive" or "negative" sentiment in the
topics.
The bad user posted tweets contain negative sentiment
on the topics against her/his ideologies and positive sentiment
on the topics she/he supports.
The clear difference between both users also is that the bad user posts negative sentiment profusely on B topics and positive sentiment on A topics.

How can I measure/show this flipping behavior in a score/value; given that each tweet is represented by a vector like: <# of Pos words, # of Neg words>.

I think a good solution will consider how dense and ideologically clear the bad user behavior.

This image summarizes the previous description:


texts: MemoryError when concatenating a large data-frame

texts:
I am creating a tool that reads CSV fields and allows the user to specify columns they want to categorize and then categorize those columns.

My issue is that these CSV files are quite large and when trying to concatenate the data-frames, my PC freezes up and I get a MemoryError.

I split the data-frame into chunks and complete the get_dummies function on each chunk and store it into a list. This works without any issues.

I then try to concatenate the entire list, as you can see in the code below.

I also delete the data-frames and the list of chunks to save up on memory.

dummies = []
columns = self.df[self.selectedHeaders]
del self.df
chunks = (len(columns) / 10000) + 1
df_list = np.array_split(columns, chunks)
del columns

for i, df_chunk in enumerate(df_list):
    print("Getting dummy data for chunk: " + str(i))
    dummies.append(pd.get_dummies(df_chunk))

del df_list
dummies = pd.concat(dummies, axis=1)

As you can see from this code, I store the columns I need and split them into chunks. Then I run the get_dummies function on each chunk and store them within a list.

When I run the concat function, I either crash or a MemoryError. If I can get the code to run and throw that error without crashing, I'll update it here.


texts:
I'm currently trying to understand regularization for logistic regression.

So far, I'm not quite sure whether I really got it. Basically, the problem is that when we add an additional features to a model we might overfit the training set. This leads to an algorithm which perfectly matches the training set, but fails for future values (as it too exactly fits what has been there in the past).

So, this is why we introduce regularization. With this we decrease the impact of the features, so the model may produce an algorithm which does a (seemingly) worse job on the training set, but overall works better.

Is this correct?


texts: Dynamic text based on filtered values in Tableau

texts:
In my dashboard, I have a polygon map of city council districts. When a certain district is filtered for on the dashboard, I want a text box to pop up on the map saying 'This district is blah blah'. I only want it for one district. When any other district is selected, no text should show up on the map.
I've tried using annotate and tooltips but those appear regardless of what district is selected in the filter.
I'm on Tableau 2020.4


texts: Estimate eps value in DBSCAN using KNN algorithm

texts:
I would like to estimate the best eps value for the DBSCAN algorithm on this dataset by following this set of rules:

Set a minPts: 10
Compute the reachability distance of the 10-th nearest neighbour for each data-point.
Sort the set of reachability distances and plot to get the elbow of the diagram (best eps value).

This is the first part of my code where I load the dataset:
import csv
import sys
import os
from os.path import join
from sklearn.cluster import DBSCAN
from sklearn.neighbors import NearestNeighbors
import matplotlib.pyplot as plt
import numpy as np

def load_data(file_path, file_name):
   with open(join(file_path, file_name)) as csv_file:
       data_file = csv.reader(csv_file,delimiter=',')
       temp1 = next(data_file)
       n_samples = int(temp1[0])
       print("n_samples=")
       print(n_samples)
       n_features = int(temp1[1])
       temp2 = next(data_file)
       feature_names = np.array(temp2[:n_features])

       data_list = [iter for iter in data_file]

       data = np.asarray(data_list, dtype=np.float64)

   return(data,feature_names,n_samples,n_features)

# --- Main program ---

file_path="Datasets/"
file_name3="CURE-complete.csv"
data3,feature_names3,n_samples3,n_features3 = load_data(file_path, file_name3)
fig = plt.figure(figsize=(8,8))
ax = fig.add_subplot(111)
fig.subplots_adjust(top=1)
ax.set_title('Dataset n. 3 of data points')
ax.set_xlabel(feature_names3[0])
ax.set_ylabel(feature_names3[1])
plt.plot(data3[:,0], data3[:,1], '.', markersize=1.2, markeredgecolor = 'blue')
plt.show()

This is where I compute the KNN-algorithm with ns (minpts) = 10.
ns = 10 #minpts
nbrs = NearestNeighbors(n_neighbors=ns).fit(data3)
distances, indices = nbrs.kneighbors(data3)
distanceDec = sorted(distances[:,ns-1], reverse=True)
plt.plot(list(range(1,len(distanceDec)+1)), distanceDec)

This is the resulting diagram, which seems unusual considering the range of values:

How can I improve my algorithm? As you can see in the following page, the plot (in my case it's reversed), should be different.
From what I understood the best eps value should be in the corner above 2.2, right?


texts: is it better to correlate and encode or encode and correlate?

texts: I have one doubt like is it better to perform label encoding and check for the correlation or should I 1st perform correlation and do label encoding? Because when I tried it both ways I'm getting different features


texts:
In the textbook im reading about Deep Learning, i found :

  When designing features or algorithms for learning features, our goal
  is usually to separate the factors of variation that explain the
  observed data [...]
  the factors are usually not combined by multiplication.

What's the meaning of the bold phrase ?

ps: what's your definition for "factors of variation" ?


texts: What is the differences between normal equation and gradient descent for polynomial regression

texts:
I'm new to machine learning and willing to study and work with machine learning. It just that I still don't get to understand the benefits of using the normal equation in some occasion in comparison with gradient descent. I use Andrew Ng's course on Coursera but the notation really makes me a hard time to understand.

I want to know more about the derivation of the cost function $J(\theta)$ for polynomial regression and the reason why he uses the transpose of vector $x(i)$


texts: Data augmentation for the inputs of CNNs to identify flowers

texts:
I want to make a neural network to identify flowers from images like this:

or similar images e.g as on https://www.pexels.com/search/flowers/

I want to use a CNN for this as in https://stackoverflow.com/questions/52463538/reducing-memory-requirements-for-convolutional-neural-network

My questions are:

Which will be better: to put whole images into training set or to divide each image into 4 parts (cutting in midline horizontally and vertically)?
Will it help if I rotate/tilt these images and put them also in training set?
Will it help if I blur these images and put them also in training set?

The main aim of network is to identify which kind of flower it is. For this, it will be trained with images of about 20 flowers. Hence, classification is the objective.

Edit: we can assume there will be only one kind of flowers in one image.


texts: My Neural Network is not learning anything

texts: I try to train convolutional neural net for a classification problem. However, my neural net is not learning anything. It guesses only two labels and ignores the rest. Even though I try to train to overfit my neural net, the loss function is not decreasing at all. My I try to make my network go as deep as 12 layers of the convolutional neural net in order to overfit the subsampling data. However, it is not working. Do you have any suggestion what to look?


texts: I am getting p value 0 for my population I teven tried zscore for normalization

texts:
a1 is the columncolumn
a1
g=scipy.stats.mstats.zscore(a1)
g
stat, p = normaltest(g)
print "%.3f, %.9f" %(stat ,p)
Result  8.570, 0.00000000
p1=scipy.stats.kstest(g,'norm')
p1
KstestResult(statistic=1.0, pvalue=0.0)
h1=scipy.stats.kstest(k, 'poisson',args=(1,))
h1
statistic=0.58, pvalue=0.0


texts:
I recently finished coding my own MLP neural network in Python. To make my code easier to read, I separated the MLP, into classes; the network class, the layers class and the neuron class, where the neuron class would do all the calculations such as the activation and calculating the error signal. Each neuron also contained an array of the weights.

To verify the results of my MLP, I tried the sklearn-MLPClassifier with the same dataset, and found that sklearn was actually much faster to train than with my code.

Would my code be faster if I rewrite it with matrices? Whats the right way to code a NN and how can I get my performance to be comaparable to that of sklearn ?


texts: Keras: Struggle with loss-functions w/ additional parameters and merging them

texts:
I'm quite new to the field of data-science and fiddle around with some time-series-data, in my case stock-data. I want to learn a neural-net to forecast them, as everyone has tried to do some time I guess? :D
I'm using tensorflow/keras and python for this.
I'm not really convinced to ever get something to work, but I want to try. Doing this, I'm stuck with my loss-function. It's mainly about "is this even right" on the one hand and it's a question on "how to get it work" on the other.
Context
In my case, I have the last n bars (open, min, max, close for each) as an input, for this illustration say it's: [[1,1,2,2], [2,2,3,3]]
The output should be a suitable "stop-loss" and "limit", so mathematically I want the system to generate the value that probably stays a bit under the interval the next bars are in (in case of "stop-loss") and the value that stays in this interval but is as high as possible (in case of "limit").
Let's say this output has a format like [3,4]
Now regarding the loss-function I got problems with.
Defining a custom loss-function takes the output-format twice, one is the predicted value ("y_pred") and one is the expected value ("y_real"). This is OK for algorithms that calculate the distance from one to another in a let's say "classical" way.
My idea (ok, guess many had this before...) is to use the real profit/loss (in financial terms) to implement the loss (in terms of training neural nets).
This implies different things and introduces my questions:

I need to get additional complex data (i.e. the future bars to detect whether my stop or loss is triggered first) in my loss-function. Technically I know how to do that, but is this the right direction?
The additional data has a different format than y_pred. It's both lists and they got the same size, but the one contains more complex input-like data of the future bars and the other contains output-like data, for both see code-snipplets above or example below). Can you tell me how to merge them? I could do that with lists, but since I got tensors I have no idea how to iterate over and merge them one-by-one. Mapping seems interesting, but then I get many more pairs than I need. See example (*) below.
The loss function will only be differentiable in parts, because I need to have some "switches" since one point up or down can make a huge difference (if my limit is exactly the maximum, then my profit is maximized, but if it's only a bit higher, then it's not reached and I have got huge losses. Will the loss-function do what I want? Is it some kind of obvious for you experts that it e.g. won't ever convergate or will be stuck in some silly valley?

Example
(*) Here is my example for illustration:
Input
[
  [
    [1,1,1,1],
    [2,2,2,2],
    [3,3,3,3]
  ],
  [
    [2,2,2,2],
    [3,3,3,3],
    [4,4,4,4]
  ],...
]

Expected output
[
  [3,5],
  [4,6],...
]

Additional data for loss-function (future of input-bars)
[
  [
    [4,4,4,4],
    [5,5,5,5]
  ],
  [
    [5,5,5,5],
    [6,6,6,6]
  ],...
]

The format I want to get
[
  [
    [
      [4,4,4,4],
      [5,5,5,5]
    ],
    [3,5]
  ],
  [
    [
      [5,5,5,5],
      [6,6,6,6]
    ],
    [4,6]
  ]
]

Hope you understand my issue and would appreciate if you took the time to give some feedback on it.
Best wishes,
Dennis


texts: Spectral clustering result interpretation

texts:
I cluster a toy data set into three groups with spectral clustering. Please see the code below. affinity='rbf' by default.

from sklearn.cluster import SpectralClustering
import numpy as np

points = np.array([[5,6], [3,3], [2,2], [7,3], [8,3], [5,4], [5,5], [9,3], [7,1]])

spectral = SpectralClustering(n_clusters=3)
labels = spectral.fit(points).labels_

The output is three clusters with labels [1 2 2 0 0 1 1 0 2] shown on the image. Why point 8 happens to be in the same cluster with 1 and 2, not with points 3,4, and 7?

UPDATE:

Affinity matrix:

Spectral embedding result:


texts: How do we decide whether mean absolute error or mean square error is better for linear regression? Are there other loss functions that are commonly used for linear regression?


texts: Classification problems - Finding Target variable outliers in data

texts:
We have Z-score, IQR etc to identify outliers in data. This could be used to eliminate outliers even in labels. For e.g. if the target variable is a housing price, we could use inter-quartile ranges to isolate outliers.
I am looking at an algorithm or a programming practice to isolate outliers in target variable data for classification exercises. I feel this is pertinent because training data could have samples with incorrect classifications. Maybe the source of error was manual oversight or maybe a freak sequence of events which normally could not have occurred. So eliminating these rows should ideally enable the algorithm to perform better classifications. Any help is appreciated


texts: ML algorithm where variable importance depends upon other variables - specifying conditionality

texts:
I am designing an algorithm that can detect cheating in a game of chess.

I have a database of players who have been flagged by moderators as cheating, along with a number of game-level characteristics such as move times and move accuracy (as determined by a chess engine) for both black and white.

Let's specify a simple regression:

cheatflag ~ whiteMoveTime + blackMoveTime + whiteMoveAccuracy + blackMoveAccuracy

Let's say cheatflag is a factor-level variable that is 0 if no cheater, 1 if white is the cheater, 2 if black is the cheater.

The issue is that the predictive value of variables like whiteMoveTime and whiteMoveAccuracy depends crucially on who is doing the cheating.  If white is cheating, those variables should have different coefficients than if black is doing the cheating.  As far as I know, running most blackbox algorithms will derive coefficients that average across all the cases of cheating without understanding the hierarchical nature of the variables.

This problem has a Bayesian flavor with some kind of conditional assumption -- if black is cheating, then these variables matter; if white is cheating, these variables matter, and so on.

How can I deal with this issue?


texts: Dimension reduction techniques in R that do not use the full distance matrix

texts:
I try to apply non-linear dimension reduction in R. As usual in machine learning I have a large data set (100 K rows). I tried the packages
RDRToolbox and vegan:

library( RDRToolbox )
res.lle = LLE(as.matrix(temp), dim = 3 , k = 10)
res.iso = Isomap(as.matrix(temp), dim = 3 , k = 10)

library( vegan )
isomap(dist(temp), ndim=10)

The first algoritms use the distance matrix internally and crash as they want to allocate 32GB. The second approach needs the distance matrix and crashes too.

Are there any tricks to do non-linear dimension reduction either with these algorithms? Or are there packages that do not use the raw and full distance matrix? Such approaches have to crash in my setting on my machine.

The only algorithm that works for me is Rtsne that appearantly does not use the whole distance matrix. largeVis was too slow so far to let it finish.


texts: How to convert RGB to One Hot encoding of Pixel in Pascal VOC Dataset?

texts:
I am trying to implement Semantic Segmentation on PASCAL VOC 2007 Dataset using Fully Convolutional Network. My Network outputs images of (Height, Width, Classes); but the training label masks are of dimension (Height, Width, 3). How do I convert the training label mask to the dimension (Height, Width, Classes) so that proper loss can be computed?
I tried to extract out unique Pixel values of the image shown below. If the dataset contains 20 Unique classes then there should be 20 Unique values but I am getting around 380 unique values for one Image. Why?? What is the correct way of getting one hot encoded value of a pixel?
img = np.array(cv.imread(AddressofPngImage))
img = tf.cast(tf.image.resize(img,[274,274]),dtype = np.int32)
img = tf.reshape(img,[-1,3]).numpy()
np.unique(img,axis = 0)

array([[  0,   0,   0],
   [  0, 192, 128],
   [  1,   1,   1],
   ...,
   [191, 222, 222],
   [191, 223, 223],
   [192, 224, 224]])


texts: Handling near duplicate observations in a regression / Bayesian model

texts:
I am working on a model where the underlying data is inherently correlated by groups.  So some of my observations are almost duplicates but not quite.

The problem is pretty simple,  I have a y variable to predict from a discrete x variable and several other potential predictor variables which may or may not be significant.   The observations are not quite independent, they're taken from groups of underlying events but I want to handle this better.

I could approach the problem by only selecting one observation from each underlying group but I think this would lose a lot of data and statistical power.  I would like some method where I can keep all my datapoints but possibly impose some artificial weight scheme or something to weight data less if there are near-duplicates elsewhere or if they are from the same underlying event.

In particular, I'm using a Bayesian regression approach with pymc3 so could do some fudging of variances in my likelihood to fit less for values which have lots of close duplicates from the same underlying group.

Does anyone have experience with similar problems and solutions to deal with it?
I think my model is currently very much overfitting due to high correlations between some predictors and predictions.


texts:
In the DNN literature, is there analysis or a term on a dropout ratio (oppositely-)proportional to the depth of a layer?

By intuition, I'd like to dropout fewer neurons on the layers next to the input and drop more when approaching the end layers. For example, passing from a p_keep = 0.8 to p_keep=0.5, especially when the inputs features are all relevant. Where p_keep is the probability of keeping a node.

For instance:

Layer1: p_keep=0.9
Layer2: p_keep=0.7
Layer3: p_keep=0.6
Layer4: p_keep=0.55
Layer5-n: p_keep=0.5


texts: Data science degree or Computer Science degree?

texts:
I'm going to be applying for college soon and some of my choices have a data science undergrad. I've taken some online courses on introductory data analysis and have been doing kaggle projects for awhile, so I know it's something I enjoy. The only questions I have are:

How relevant is the curriculum? (For example, the UCSD data science major)
How reliable is the degree in the upcoming future? (Would I be able to easily get a job like if I had a CS BS)
Am I better off doing CS and minoring in stats or something else?

I am also planning on getting either a masters or PhD in DS/ML. I don't know if it would better to get a lower level understanding of computers like I would probably get in a CS curriculum and then just focus on data science in grad school.

Thanks!


texts:
I'm trying to compute the information content of a function that reranks a list. Perhaps more precisely, I'm trying to compare the information content of different arguments to that function.

Suppose that I have a scoring function, S(i,n) that takes and item, i, and gives is a score of, say, 0-1, where n is some sort of parameter that we'll get back to in a moment. I can run this on a set of items, i*, and thereby (ignoring ties) rank these items S(i*,n1) -> {s1, s2, s3 ...} where these scores correspond to the items, and so can be ordered by simple sorting to rank the items. (It's possible that the actual scores are enough, that we don't need the rank ordering, in what is to follow.)

The scoring function depends on n, so S(i*,n1) usually != S(i*,n2) (Again, ignoring the unusual case of a tie.) What I'm interested in is measuring the "information utility" of the ns. So, for example, in the (neglected by definition) case where so S(i*,n1) == S(i*,n2), n2 provide no additional information over n1 (and v.v.), but in the more common case where S(i*,n1) != S(i*,n2) the scores, and perhaps the rankings will differ, and so n2 provides some more information over n1 (and v.v.)

(Note that the reason I keep harping on the ranking and not the scores is that there is an infinitude of cases where the scores will change by the rankings NOT change! I really care a more about the rankings than the scores. Say, for example, I'm trying to choose the best drug, or the best stock, or something like that. There which comes in first matters more than the specific scores.)

There are, of course, tests, like the Wilcoxon's and Spearman's that tell me on a Z scale how much the rankings differ under different ns. And perhaps the simplest thing to do is just to pick the largest n and call the most informative n the one that gives the largest Z on one of those tests.

But is there a well understood way to directly measure the information change between two lists, that would seem to be a more theoretically well motivated way of doing this.

Thanks in advance for your suggestions.


texts:
I am trying to predict on my test set and I get the following error
AttributeError: 'Functional' object has no attribute 'uses_learning_phase'

The model is created from the Class Model imported from from keras.engine import Module
All the module I imported are from tensorflow.keras except for the package mentioned above.
I also tried Importing everything from Keras alone but I still get the same error !
I have tensroflow 2.0 and keras 2.4.3


texts: Linear regression with white Gaussian noise

texts:
I am new to machine learning , so this question may sound fundamental. My task is to estimate the parameter vector of the equation with the least squares method:

$y = \theta_0 + \theta_1x + \theta_2x^2 + η$

Where η corresponds to white Gaussian noise with mean 0 and variance 0.1

Also , I have been given the prior values of the parameter vector , say [-1,0.3,0.6] . I have to generate the N points of the training set .

Should regresor be like this :

[1 0.1 (o.1)^2] (for x = 0.1)

And should I calculate a priori results by adding noise


texts: What's the best graphic in R to study air pollution?

texts:
I have a Python dataframe (in addition to an excel file) with data on air pollution for countries, (one line every day between 2014 and 2020). I would like to see what's the polluant the most related to industrial production, fort that I want to compare the data during quarantine and after.

These are the first lines of my excel file (also a python dataframe)

I am new to Python/R as well as statistics and I do not really know how to handle this data and what type of graphic I should use ?

Any information is welcome

Thank you very much


texts:
A commonly heard sentence in unsupervised Machine learning is

  High dimensional inputs typically live on or near a low dimensional
  manifold

What is a dimension? What is a manifold? What is the difference?

Can you give an example to describe both?

Manifold from Wikipedia:

  In mathematics, a manifold is a topological space that resembles
  Euclidean space near each point. More precisely, each point of an
  n-dimensional manifold has a neighbourhood that is homeomorphic to the
  Euclidean space of dimension n.

Dimension from Wikipedia:

  In physics and mathematics, the dimension of a mathematical space (or
  object) is informally defined as the minimum number of coordinates
  needed to specify any point within it.

What does the Wikipedia even mean in layman terms? It sounds like some bizarre definition like most machine learning definition?

They are both spaces, so what's the difference between a Euclidean space (i.e. Manifold) and a dimension space (i.e. feature-based)?


texts: Maximization problem with time-varying choice variables

texts: I am looking at building a reinforcement learning problem where the objective function is say, profit. Agents in this setup can make different choices, depending on the time of interaction with the principal. Is there a model I can refer to for this?


texts: Logistic loss increasing while training with minibatches using the adam algorithm

texts:
I am trying to write my own code to use the adam algorithm for logistic regression. I am pretty sure It is training correctly as when I run it I am able to accurately classify a bunch of toy data that was generated using the parameters from the model. However, the cost keeps going up after each epoch and I don't think that is supposed to happen. Here are the two functions where I think there could be a problem:

def log_loss(mX, mY, params, penalty = 'l2', C = 1.0):

    #make sure y is the right shape
    mY = mY.reshape(-1,1)

    # get model parameters from dictionary
    w = params[list(params.keys())[0]]

    intercept = len(params.keys()) == 2

    if intercept:
        b = params[list(params.keys())[1]]

    else:
        b = 0

    mm = mX.shape[0]

    # compute model hypothesis
    z = mX.dot(w) + b
    h = sigmoid(z)

    # compute gradient
    error = h - mY
    dw = C * np.dot(np.transpose(mX), error) / mm

    if intercept:
        db = C * np.sum(error) / mm

    else:
        db = None

    # compute cost without regularization
    log_loss = np.multiply(y, np.log(h)) + np.multiply(1 - y, np.log(1 - h))
    cost = -C * np.sum(log_loss) / mm

    # now add regularization
    if penalty == 'l2':
        dw += w / mm
        cost += np.sum(np.multiply(w, w)) / (2 * mm)

    elif penalty == 'l1':
        dw += np.divide(np.abs(w), w)
        cost += np.sum(np.abs(w)) / (2 * mm)

    if db:
        grads = {'dw': dw, 'db': db}

    else:
        grads = {'dw': dw}

    return grads, cost

I am reasonably sure that the cost function implementation is correct. The function that trains the model is here:

def train_model(X, y, lr = 0.01, mini_batch_size = 64, beta1 = 0.9,
                beta2 = 0.999, epsilon = 1e-6, num_epochs = 1000,
                print_cost = True, fit_intercept = True,
                penalty = 'l2', C = 1):

costs = []
t = 0
m, n = X.shape

params = initializeParameters(n, fit_intercept)
v, s = initialize_params_adam(n, fit_intercept)
num_mini_batches = math.ceil(m / mini_batch_size)

for i in range(num_epochs):

    minibatches = random_mini_batches(X, y, mini_batch_size)
    epoch_cost = 0

    for minibatch in minibatches:

        (mX, mY) = minibatch

        grads, cost = log_loss(mX, mY, params, penalty, C)

        epoch_cost += cost

        t += 1
        params, v, s = update_params_adam(params, grads, v, s, t,
                                          lr, beta1, beta2, epsilon)

    if print_cost:
        costs.append(epoch_cost / num_mini_batches)
        print("Cost at epoch %i: %f" %(i + 1, epoch_cost / num_mini_batches))

return params, costs

When I run the full program I get this output:

Cost at epoch 1: 15593.325359

Cost at epoch 2: 9042.439993

Cost at epoch 3: 9616.118761

Cost at epoch 4: 10005.095845

Cost at epoch 5: 10436.960861

Cost at epoch 6: 10914.245428

Cost at epoch 7: 11425.692792

Cost at epoch 8: 11959.885890

Cost at epoch 9: 12507.349951

Cost at epoch 10: 13060.884386

Cost at epoch 11: 13615.315131

Cost at epoch 12: 14167.092496

Cost at epoch 13: 14713.897392

Cost at epoch 14: 15254.310444

Cost at epoch 15: 15787.553480
...

and yet the model perfectly classifies the toy data. What is going on?

EDIT: I just tried lower the learning rate and that definitely seemed to help, but it still goes down for a while and then slowly goes up a bit. Could this be because the learning rate is still too large When we are near the minimum?


texts: When using Absolute Error in Gradient Descent, how to calculate the derivative?

texts:
What is the derivative of the Loss Function (Absolute Error) with respect to the feature weights that is used to update the weights?

Couldn't find anything specific about it anywhere.


texts: how Lasso regression helps to shrinks the coefficient to zero and why ridge regression dose not shrink the coefficient to zero?

texts:
How Lasso regression helps feature selection of model by making the coefficient to zero? ,
I could see few below with below diagram ,can any please explain in simple terms how to corelate below diagram with i.) how lasso shrinks the coefficient to zero ii.) how ridge dose not shrinks the coefficient to zero


texts: What is the procedure to realize zero mean and unit variance?

texts:
The tag on feature scaling says:

  Popular feature scaling types include scaling the data to have zero
  mean and unit variance, and scaling the data between a given minimum
  and maximum value.

Are there any details that help us to generate zero mean and unit variance?


texts:
What is the fully-convolutional model?

Is fully-convolutional model a model that has only convolutional layers (with Batch-norm and Activation) and has not any: max-pool, fully-connected, and other layers?

If it is true, then why Yolo v2 neural network for object detection is named Fully-Convolutional model if it uses Max-pool layers: https://arxiv.org/pdf/1612.08242v1.pdf

  Table 2: The path from YOLO to YOLOv2. Most of the listed design
  decisions lead to significant increases in mAP. Two  exceptions are
  switching to a fully convolutional network with anchor boxes and using the new network.

Also why Fully Convolutional Networks for Semantic Segmentation is named Fully Convolutional if it has Max-pool layers: https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf


texts: Is there any library available for balancing imbalanced text dataset?

texts: I have a text dataset similar to newsgroup dataset, the problem with the dataset is that it is highly imbalanced. So is there any readily built library that will do upsampling or downsampling with a function call?


texts: Not able to interpret decision tree when using class_weights

texts:
I'm working with an imbalanced dataset. I'm using a decision tree (scikit-learn)  to build a model. For explaining my problem I've taken iris dataset.
When I'm setting class_weight=None, I understood how the tree is assigning the probability scores when I use predict_proba.
When I'm setting class_weight='balanced', I know its using target value to calculate class weights but I'm not able to understand how the tree is assigning the probability scores.
import sklearn.datasets as datasets
import pandas as pd
import numpy as np
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split

from sklearn.externals.six import StringIO
from IPython.display import Image
from sklearn.tree import export_graphviz
import pydotplus

iris=datasets.load_iris()
df=pd.DataFrame(iris.data, columns=iris.feature_names)
y=iris.target

X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.33, random_state=1)

# class_weight=None
dtree=DecisionTreeClassifier(max_depth=2)
dtree.fit(X_train,y_train)

dot_data = StringIO()
export_graphviz(dtree, out_file=dot_data, filled=True, rounded=True, special_characters=True, feature_names=X_train.columns)
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
Image(graph.create_png()) # I use jupyter-notebook for visualizing the image

# printing unique probabilities in each class
probas = dtree.predict_proba(X_train)
print(np.unique(probas[:,0]))
print(np.unique(probas[:,1]))
print(np.unique(probas[:,2]))

# ratio for calculating probabilities
print(0/33, 0/34, 33/33)
print(0/33, 1/34, 30/33)
print(0/33, 3/33, 33/34)

The probabilities assigned by the tree and my ratios (determined by looking at tree image) are matching.
When I use the option class_weights='balanced'. I get the below tree.
# class_weight='balanced'
dtree_balanced=DecisionTreeClassifier(max_depth=2, class_weight='balanced')
dtree_balanced.fit(X_train,y_train)

dot_data = StringIO()
export_graphviz(dtree_balanced, out_file=dot_data,filled=True, rounded=True, special_characters=True, feature_names=X_train.columns)
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
Image(graph.create_png())

I'm printing unique probabilities using below code
probas = dtree_balanced.predict_proba(X_train)
print(np.unique(probas[:,0]))
print(np.unique(probas[:,1]))
print(np.unique(probas[:,2]))

I'm not able to understand (come-up with a formula) how the tree is assigning these probabilities.


texts:
I am working on multilingual (English, Arabic, Chinese) NER and I met a problem: how to tokenize data?
My train data provides sentence and list of spans for each named entity.
e.g.

[('The', 'DT'),
 ('company', 'NN'),
 ('said', 'VBD'),
 ('it', 'PRP'),
 ('believes', 'VBZ'),
 ('the', 'DT'),
 ('suit', 'NN'),
 ('is', 'VBZ'),
 ('without', 'IN'),
 ('merit', 'NN'),
 ('.', '.')]

[('你', 'PN'),
 ('有', '.'),
 ('没', 'AD'),
 ('有', '.'),
 ('用', 'VV'),
 ('过', '.'),
 ('其它', 'DT'),
 ('药品', 'NN'),
 ('？', 'PU')]

What the best way to tokenize input data? There are main different alternatives I consider: word level, wordpiece level, BPE.
BPE does not work with Chinese and Arabic because of unicode thing. I doubt about word level because I am not sure what is a word in Chinese.
What can you recommend me?


texts: Using LSTMs for modelling and forecasting several time series generated by the same process

texts:
My scenario is roughly the following:

Imagine 500 cars, all Toyota Corollas (or whatever). While these cars have many similarities, they are not exactly identical: some of them have 1.5 liter engines while others have 1.8 liter engines, and some have a high mileage while others have barely left their garage. Still, we can imagine that all cars are the product of one process with a bunch of parameters. Some of these parameters are observed, while others are unobserved.

I have data on the condition of each car at a set of times t_i. For our purposes, "condition" is represented by a number between 0.0 and 1.0, where 0.0 means "as good as new" and 1.0 means "basically a wreck". My goal is to build a model which lets me forecast the condition of each car at some future time, based on its condition at the previous time steps as well as its "parameters" like mileage and engine size. I could train a separate model for each of the cars, but I likely don't have enough data for each car for this model to provide accurate forecasts.

I've already made an attempt at this using ordinary machine learning methods by transforming the full history of each car into one data point by encoding the condition at each of the previous time steps as a feature. My results are kind of so-so, and I'd like to try out LSTMs for this purpose.

However, I have a hard time figuring out how to do this with LSTMs. A bit of googling has led me to lots of tutorials on how to use LSTMs to forecast two properties of the same "object", say, condition and mileage for a single car, but none on how to predict the same property for a bunch of "different objects" with additional parameters. I assume just plugging

[conditionCar1, mileageCar1, colorCar1, conditionCar2, mileageCar2, colorCar2, (...)]

observed at a lot of time steps into an LSTM model wouldn't work very well. Is what I'm trying to do viable, and if so does anybody have any pointers to relevant information?


texts: Can I fine-tune the BERT on a dissimilar/unrelated task?

texts:
In the original BERT paper, section 3 (arXiv:1810.04805) it is mentioned:
"During pre-training, the model is trained on unlabeled data over different pre-training tasks."
I am not sure if I correctly understood the meaning of the word "different" here. different means a different dataset or a different prediction task?
For example if we pre-train the BERT on a "sentence-classification-task" with a big dataset. Then, should I fine-tune it again on the same "sentence-classification-task" task on a smaller and task-specific data-set or I can use the trained model for some other tasks such as "sentence-tagging"?


texts:
Is there a way to weight instances based upon the values of a feature?
For instance for making a sample better resemble a population on certain parameters.


texts:
I have date and time in one column of the form
 2004/12/15 9:30 , how can I format it  to date, so that I can plot it on x axis. Also posxlt is not working for me.


texts: Generative Model for learning Periodic solutions 3-body/N-body problems

texts:
I am tasked with finding research where a GAN or any other generative model is used to generate new shapes of 3-bodies moving under the influence of each others' gravitational pull, in a periodic motion, like for example here : with the 3-bodies going about in that 8-shaped orbit . My findings tell me that, this is a "Periodic solution for a 3-body problem ", and I learned that there are a finite number of these solutions out there. I also learned that a restricted 3-body problem might have more periodic solutions per se. I am not too sure on the authenticity of that claim in the previous line.

I was wondering if one could make a generative/GAN-like model to make it learn the distribution of these shapes, and thereby generate more and more of these shapes? If so, has there been research on this already? If yes, could someone point me to it, since i am unable to do so myself, coming from a pure Machine Learning-Software Development background, not having much idea of the terms like Lagrangians and the Eulerians and so on.

P.s. 3-body is just for reference, i dont mind hearing about anything similiar on the N-body problem front as well.

So please help me community. Any and every ideas/questions/intuitions are appreciated. :-)


texts: Visualizing F-score differences in information extraction

texts:
I have several corpora and NLP systems (including a few merge ensembles of output of these systems combined in unions and intersections) with which I have extracted the annotation span sets {(begin, end)} for each corpus across all documents within the corpus and compared the span sets to each corpus's respective gold standard and thus obtained standard measures of F-score, precision and recall.

I am trying to qualitatively assess why certain systems don't perform as well as a particular ensemble combination on F-score, so I figured the easiest way would be to generate precision-recall or ROC curves.

The task is just a simple binary classification: either a span of text is annotated (labeled as 1) or it is not (labeled as 0).

I have numpy vectors of the same length for each document in the corpus for both the system predictions and the gold standard, so I plan on using these for y_true and y_predict when trying to generate my ROC curve.

Is this a good approach to observe the behavior of my F-scores, assuming I plot them all on the same graph? If not, any recommendations for a better approach would be most appreciated.


texts: Place of backward and its relation with batches in PyTorch

texts:
I am implementing a dependency parsing model using PyTorch and little bit confused about the situation that I explained below. When calculating loss and backward the model; I tried different things.

When I use the code below exactly, and make batch size 1 (1 batch in iteration):

Loss looks like decreasing, however the predictions are not getting well after 20 epochs.

When I use the code below exactly, and make batch size 100(0):

I get an error: RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
As the error message said, when I use inside_loss.backward(retain_graph=True) instead of inside_loss.backward(), the execution tooks too long. (I did not get result yet)

When I comment out the inside_loss and uncomment the lines after the for loop:

The loss is not changing.

The code is here:

def forward(self, x,):
# x is the scores of my vocab. It has grad so cannot change its values.
# So take the clone
x_prime = x.clone()

# loss = Variable(torch.zeros(1), requires_grad=True)
loss_v = torch.zeros(1)

for i in range(x_prime.size(0)):

    # Some operations that changes x_prime's values
    # Calculate sentence_probs and sentence_scores from x_prime's values

    eisner_values = eisner_torch(sentence_probs)

    # Changed x_prime above; so update the x
    x = x_prime

    # Get the gold dependencies
    gold_deps = return_gold_deps(i, sentence_to_dependencies)

    if gold_deps is None:
        continue

    mask = np.greater(np.asarray(eisner_values), -1)

    # Calculate hinge loss
    inside_loss = hinge(sentence_scores, eisner_values, gold_deps, mask, 1)

    # Calculate total loss in the batch
    loss_v += inside_loss.data

    inside_loss.backward()

    # Optimizer step
    if self.opt is not None:
        self.opt.step()
        self.opt.optimizer.zero_grad()

# loss.data = loss_v.data
# loss.backward()
# Optimizer Step
return loss_v

I use the Adam optimizer for this task:

model_opt = NoamOpt(model_size=d_model, factor=1, warmup=200,
                torch.optim.Adam(model.parameters(), lr=0, betas=(0.9, 0.98), eps=1e-9))

What are the problems in here? How can I solve that issue?

Thanks in advance.

Link to stackoverflow: Question also asked in here


texts: Multivariate Regression Error “AttributeError: 'numpy.ndarray' object has no attribute 'columns'”

texts:
I'm trying to run a multivariate linear regression but I'm getting an error when trying to get the coefficients of the regression model.
The error I'm getting is this: AttributeError: 'numpy.ndarray' object has no attribute 'columns'
Here's the code I'm using:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as seabornInstance
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn import metrics
%matplotlib inline

# Main files
dataset = pd.read_csv('namaste_econ_model.csv')
dataset.shape
dataset.describe()
dataset.isnull().any()

#Dividing data into "attributes" and "labels". X variable contains all the attributes and y variable contains labels.

X = dataset[['Read?', 'x1', 'x2', 'x3', 'x4', 'x5', 'x6' , 'x7','x8','x9','x10','x11','x12','x13','x14','x15','x16','x17','x18','x19','x20','x21','x22','x23','x24','x25','x26','x27','x28','x29','x30','x31','x32','x33','x34','x35','x36','x37','x38','x39','x40','x41','x42','x43','x44','x45','x46','x47']].values
y = dataset['Change in Profit (BP)'].values
plt.figure(figsize=(15,10))
plt.tight_layout()
seabornInstance.distplot(dataset['Change in Profit (BP)'])
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
regressor = LinearRegression()
regressor.fit(X_train, y_train)
coeff_df = pd.DataFrame(regressor.coef_, X.columns, columns=['Coefficient'])
coeff_df

Full error:

Traceback (most recent call last):
File "", line 14, in coeff_df = pd.DataFrame(regressor.coef_, X.columns, columns=['Coefficient'])
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

Any help on this will be highly appreciated!


texts:
I am using decision tree classifier to predict some block selected based on the below data.

I am able to predict the "Block selected" column based on the data. How to get the second best, third best prediction and so on(I need a ordered list)? Can I get this using decision tree? Or should I be using a different model?
Any thoughts on how to do this using python scikit-learn?


texts: Understanding the process for generating Random Forest model in caret

texts:
This is not so much a problem, as it is me making sure I understand what's happening with my Random Forest algorithm.
Below, I've set a few parameters. Am I right in thinking that this is the stages:

Model is being run 10 times due to 10 Fold Cross Validation, whereby data is being split into 10 folds (9 used for training, 1 for validation - repeated k times, each time with a different group used for validation).

For each model, 500 individual decision trees (ntree) are being generated to create the Random Forest.

Because tuneLength is 5, steps 1 and 2 are repeated 5 times - each time with mtry being set to a different number.

Also, I ran this on my training data.
Is it normal practice to next pass the separate test data into the model to check how well it's able to predict the target variable. Then if satisfied with the outcome, re-create the model with all of the data?
I could really use some clarification here, as I think I may be getting this all wrong. The results of the below model tell me what the optimal mtry is (i.e. 2). So I'm unsure if I should then be creating an entirely new model, removing the trControl parameter, and manually adding in mtry as 2 if possible.
set.seed(1)
rf_test <- train(mortality ~.,
                 data = rf_train,
                 method = "rf",
                 ntree = 500,
                 trControl = trainControl(method = "CV", number = 10),
                 tuneLength = 5)
print(rf_test)


texts: How to Classify Documents whether they are similar to previous documents or not?

texts: I am a newbie in machine learning.I have 100 text documents.I need to build a model on those 100 text documents and if i give new documents it has to give whether this new document is similar to those 100 documents or not.Which is the best model for this problem?


texts: Genetic Programming Python library with a Scikit-Learn inspired API

texts:
I've been working with GPlearn for the passed couple of months as part of my research, but it turns out that it provides only a binary symbolic classifier. Do you guys know any other python library which do provide non-binary symbolic classifier and is scikit-learn based API?
Thanks!


texts:
There are total 50 students(john, Roy..) and used some action to do a job. My dataSet something like this

    John    Roy    Micheal   Ron  ....... Smith
     A       B        B       A            C
     A       A        C       B            B
     C       A        A       B            B
     .       .        .       .            .
     .       .        .       .            .
     .       .        .       .            .
     F       G        E       A            G

Here A,B,C...G are strings. Final data is like this...

         A   B   C   D   E   F   G
 John    3  34  23  34   4   3   1
 Roy     5  23  12  3    5   39  46
 ...................................

This means John used "A" 3 times whereas Roy used "A" 5 times.
So based on their action I want to cluster them (i.e. the student used the same number of activities will be in the same group).

Which clustering method can be used and how ?


texts: A Neural Network That Learns Bitwise XOR

texts:
I am trying to build a deep neural network that learns the coordinate-coordinate bitwise XOR of two matrices, but it performs poorly.

For example, in the 2 bits case, its accuracy stays around 0.5. Here is the code snippet:

from keras.layers import Dense, Activation
from keras.layers import Input
import numpy as np
from keras.layers.merge import concatenate
from keras.models import Model

size=1
data1 = np.random.choice([0, 1], size=(50000,size,size))
data2 = np.random.choice([0, 1], size=(50000,size,size))
labels  = np.bitwise_xor(data1, data2)
a = Input(shape=(size,size))
b = Input(shape=(size,size))
a1 = Dense(size, activation='sigmoid')(a)
b1 = Dense(size, activation='sigmoid')(b)
merged = concatenate([a1, b1])
hidden = Dense(1, activation='sigmoid')(merged)
hidden = Dense(3, activation='sigmoid')(hidden)
hidden = Dense(5, activation='relu')(hidden)
hidden = Dense(4, activation='sigmoid')(hidden)
hidden = Dense(3, activation='sigmoid')(hidden)
outputs = Dense(1, activation='relu')(hidden)

model = Model(inputs=[a, b], outputs=outputs)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit([data1, data2], np.array(labels), epochs=15, batch_size=32)

What's going on here?

Epoch 1/15
50000/50000 [==============================] - 7s 130us/step - loss: 0.7118 - acc: 0.5044
Epoch 2/15
50000/50000 [==============================] - 4s 78us/step - loss: 0.6933 - acc: 0.5023
Epoch 3/15
50000/50000 [==============================] - 4s 74us/step - loss: 0.6934 - acc: 0.5030
Epoch 4/15
50000/50000 [==============================] - 4s 86us/step - loss: 0.6935 - acc: 0.5002
Epoch 5/15
50000/50000 [==============================] - 4s 79us/step - loss: 0.6934 - acc: 0.5015
Epoch 6/15
50000/50000 [==============================] - 5s 96us/step - loss: 0.6935 - acc: 0.5030
Epoch 7/15
50000/50000 [==============================] - 5s 105us/step - loss: 0.6934 - acc: 0.5026


texts:
I am interested in joining some Data Science Groups on Linked In.  The relevant list is long.  I don't think it is to my advantage to join all, but I am not sure how to evaluate the individual group's potential to elevate my career.

Is there some clear criteria for which I can evaluate potential groups?


texts: i have training data from 2015-2017 and testing data of 2018. i have multiple variables my data is multivariate time series data.i want to predict 2019 data by using test data of 2018.is it possible? i am confused about Long short term memory neural networks working what is actually it will do.does my problem come under multivariate multi step forecasting? or multivariate single step forecasting?


texts: Moving from macbook (without GPU) to linux system with Titan V, only getting a 4x speedup, what am I doing wrong?

texts:
I was prototyping a network architecture out on the macbook, and after finding something I was somewhat happy with, I wanted to test it out on a big data set on a system with a Titan V as the macbook was very slow for the bigger dataset (12 hours). I was expecting at least a 50x speed up per epoch over the CPU, if not more. Why might the speedup be only 4x (3 hours)?

Err it turned out to be some very tricky driver issues, and by just cleaning out my vm and starting over, was able to fix it.

This line of code in particular was invaluable - some of the other checks on "is tensorflow using the gpu" I found hovering around the web were not adequate to solving this:

import tensorflow as tf
with tf.device('/gpu:0'):
    a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
    b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
    c = tf.matmul(a, b)

with tf.Session() as sess:
    print (sess.run(c))


texts:
I'm attempting to create my own neural network optimization model from scratch. I'm getting hung up on backpropagation.

I have just a few basic questions:

When adjusting a given weight by the learning rate, $\alpha$, the weight is selected by changing the weight with the highest $\delta$. Are these deltas all calculated simultaneously and independently? Or is it calculated one layer at a time, or some alternative method?
How is the delta calculated? I am not following any of the calculations since it seems to leave delta unsolved. Maybe I'm misunderstanding. Is it common to use finite differencing to estimate deltas, or is this very slow/inaccurate?
When optimizing on time series training data, is the total error the sum of the output errors of all the observations? If so, is the delta the change in the total error? What is the most efficient way to approach this problem?

If this is inappropriate for this board, apologies in advance.

Edit: Clarifying my question:

Suppose we have 3 inputs, two layers of hidden neurons with Sigmoid activation functions, and 3 neurons per layer, and one output neuron.

I will denote weights as $w_{i,j,k}$ where $i =$ target layer, $j =$ target node, and $k =$ preceding node. Nodes will be denoted as $n_{x,y}$ where $x = $ layer and $y =$ node.

The error for the output is given as $E = \Sigma \cfrac{1}{2} (Target - Out)^2$.

I want to clarify that I am doing this properly. Calculating $\delta$ is simple if the weight targets the output layer. However, if I want to find $\cfrac{\partial E}{\partial w_{2,1,1}}$, is it expanded as:

$\cfrac{\partial E}{\partial Out} \cfrac{\partial Out}{\partial net_O}
\cfrac{\partial net_O}{\partial n_{2,1}} \cfrac{\partial n_{2,1}}{\partial net_{2,1}} \cfrac{\partial net_{2,1}}{\partial w_{2,1,1}}$ where $Out = \cfrac{1}{1+e^{-net_O}}$

$\cfrac{\partial E}{\partial Out} = (Out - Target)$

$\cfrac{\partial Out}{\partial net_O} = (Out)(1-Out)$

$\cfrac{\partial net_O}{\partial n_{2,1}} = w_{3,1,1}$

$\cfrac{\partial n_{2,1}}{\partial net_{2,1}} = n_{2,1}(1-n_{2,1})$

$\cfrac{\partial net_{2,1}}{\partial w_{2,1,1}} = n_{1,1}$

Putting it all together:

$\cfrac{\partial E}{\partial Out} \cfrac{\partial Out}{\partial net_O}
\cfrac{\partial net_O}{\partial n_{2,1}} \cfrac{\partial n_{2,1}}{\partial net_{2,1}} \cfrac{\partial net_{2,1}}{\partial w_{2,1,1}} = $

$(Out - Target)(Out)(1-Out)(w_{3,1,1})(n_{2,1})(1-n_{2,1})(n_{1,1})$

In essence, to calculate $\delta$ for each layer of weights further back in the back propagation, you just add two more terms, being the derivative of the sigmoid for that node, and the node previous, as well as another weight term? Is this correct?


texts: How to replace words in a sentence with their POS tag generated with SpaCy efficiently?

texts: How is it possible to replace words in a sentence with their respective PoS tags generated with SpaCy in an efficient way?


texts: How to consider categorical variables in distance based algorithms like KNN or SVM?

texts:
For example lets say I have a dataset with independent features age, gender, name, and income. While my dependent variable is load approval status.

If I want to use KNN or SVM, do I need to convert gender into a numerical variable? and if yes, how is it used in algorithms such as KNN or SVM which are distance based algorithms?