texts: Unsupervised clustering without of Data which is supposed to be on a linear function

texts:
When I have a dataset where each datum has x and y,
and the (x,y) has a relation of one of y = a_i*x + b_i (i=1,2,...).

Is the process written below available? and which algorithm does it belong to?

The process is.....

I have many points (x, y).
The machine finds 2 linear functions which represents the points.
The machine eliminates points which are far from the 3 lines.

In this case, I think I put parameters the number of linear functions and criterion to judge if a point is on a line or not..

The first figure is my dataset.

I want to have a machine which accepts the number of line (3 in this case)
and finds 3 lines(as the second figure (the lines in the figure are just put for idea without computation)), and then finally suggests points which may belong to neither of them. (In this case, for example, (71.6, 22))

Should I, for instance, extend the k-means algorithm to achieve this procedure?///


texts: How to determine layers structures appropriate for a specific application?

texts:
I'm playing around with Keras. I went through their example of MNIST data, cats vs dog etc.
The structure of the network they suggest is different in each case. The MNIST stucture:
# Build a simple model
inputs = keras.Input(shape=(28, 28))
x = layers.experimental.preprocessing.Rescaling(1.0 / 255)(inputs)
x = layers.Flatten()(x)
x = layers.Dense(128, activation="relu")(x)
x = layers.Dense(128, activation="relu")(x)
outputs = layers.Dense(10, activation="softmax")(x)
model = keras.Model(inputs, outputs)

So basically

preprocessing
flattening the data
2 intermediate layers
output layers.

The cats vs dogs classifier has some preprocessing as well, but then many more intermediate layers of different types. I've played around with each, adding/remove layers and changing characteristics in each. Seems to influence loss/accuracy, but then many different variations also seems to have pretty similar results.
My question is - for some lambda application, what is a good process to determine the basic architecture of the networks (how many layers, what types)? I understand that Keras has a tuner to set the parameters, however my understanding is that this is for the model parameters. I'm talking about one step back from that - how many layers? Which types? Are there good rules of thumbs for that, good resources of "standard" structure as a function of types of inputs?


texts:
I just saw this video on youtube: https://www.youtube.com/watch?v=kopoLzvh5jY&t=9s
Which Algorithm did OpenAI used to create a hide and seek playing Agent?
Was it Genetic Algorithm or Policy Gradients or something else?
If it was Policy Gradient method, then which Policy Gradient method did they used?


texts: How to proceed 2 executions in 1 step in hive?

texts:
I am wondering if there is a way to proceed 2 exectuions in 1 step in hive.
For example:

SELECT * FROM TABLE1
SELECT * FROM TABLE2
;

Do this in one window, and do not have to open 2 hive windows to execute each line separetly.

Can it be done on HUE?


texts: Google team drive and google colaboratory uploading/downloading files

texts:
So I have google Team drive set-up with all the permissions.
1. How I can access google team drive in google colaboratory? I think i'm missing something... it's always showing me output of my drive
2. I want to download all the "data" from google team drive to my google colab notebook is it possible to make it automatically?


texts: Selecting which categorical variable optimizes a target variable?

texts:
this is my first question here!
I am working on developing a machine learning model that can select which NFL play type (pass, run, punt, or kick) will optimize win probability added based on a variety of game conditions.
I can train models for each of the 4 types of plays that can predict a value for wpa based on game conditions.
What I would like to be able to do is train a model that will evaluate the predicted wpa for each of the 4 options and select the option with the highest win probability.
I'd like the input to be a set of game conditions (i.e. 4th and 3 at the 37-yard line trailing by 6 with 4:51 left in the 4th quarter), and the output to be the type of play (pass, run, punt, kick) as well as the predicted wpa.
So far, my thought is to build a regression model for each of the play types separately, and then write code to take the input values and plug them into each of the four models to predict wpa and then my code will have to select the play associated with the highest wpa.
It would be simple to build a model that could predict the most likely play given game circumstances (input: 4th and 3 at the 37-yard line trailing by 6 with 4:51 left in the 4th quarter, output: punt). I understand that in this situation, play type is the target variable and i would build a classification model to predict play type.
What's confusing me is now my target variable seems to be play type, but my objective function is maximizing wpa. Are there machine learning models that can handle this without the need for discrete models for each play type?
Thanks!
-Neal


texts: What machine learning algorithms to use for unsupervised POS tagging?

texts:
I am interested in an unsupervised approach to training a POS-tagger.

Labeling is very difficult and I would like to test a tagger for my specific domain (chats) where users typically write in lower cases etc. If it matters, the data is mostly in German.

I read about about old techniques like HMM, but maybe there are newer and better ways?


texts: In recommender systems, how to avoid recommending a product that the user has just bought?

texts:
Suppose I'm running an online store that sells many products, but from only a couple of categories, say: $A$, $B$, or $C$.

Let's say a user has bought a product in the A category, and there's no sense of recommending to him products from that category, at least for some time. This might be in a form of a "you might also like" panel, or indeed a modification to the recommendation engine itself.

Even better: let's suppose that I know the probability density $P(\Delta t | A)$ of a product being re-bought after time $\Delta t$ from within the $A$ category (same for other categories).

My question is: how to choose the time threshold $\tau_X$ for each category $X\in \{A, B, C\}$ such that once a product in category $X$ was bought by a given user, other products from that category are not going to be recommended to that user for time $
\tau_X$?

The main "gain" from not showing a user products from that category is that we can use the recommendation slots for other categories, thus increasing the chance of the user buying new products (from other categories).

Does this set up resemble any well-known problem? What keywords should I google to find a more formal problem statement and/or possible solutions?


texts:
I am beginner in data-science. I am trying to understand this PyTorch code for gradient computation using custom autograd function:

class MyReLU(torch.autograd.Function):

@staticmethod
def forward(ctx, x):

    ctx.save_for_backward(x)
    return x.clamp(min=0)

def backward(ctx, grad_output):

    x, = ctx.saved_tensors
    grad_x = grad_output.clone()
    grad_x[x < 0] = 0
    return grad_x

However, I don't understand this line : grad_x[x < 0] = 0. Can anyone explain this part?


texts: Is recall more important than precision for mass mailings?

texts:
Say for example, I built a classification model for a mailing campaign that will be applied to 1M records. The positive class for the model would be customers and the negative records would be non-customers.

For the mailings, we select the top 100,000 scoring (highest predicted probabilities) records. We are expecting a low response rate but we want to make sure we send mail pieces to all of the highest predicted prospects.

We bin the model scores into deciles and only select from decile 1 (highest scoring) which we'll say is >= 0.10 predicted probability. The model has a 95% recall and 5% precision at this threshold.

Does it make sense in this scenario to get the highest recall possible? Or would precision be important here and I should optimize on that or just optimize them both together.

The goal is to get the most customer conversions within that 100k and we already know there will be a good amount of false positives but we want as many true positives as possible.


texts: To datawarehouse or not to data warehouse?

texts:
I was wondering if you will be as so kind to assist me with a quick question (will to be happy to explain more if you are willing to...). I am researching and setting up a system to do a machine learning job (training) to find correlations between Social Media (or other digital trails from wearables etc.) information of a user and his scores on personality tests.

The scores are in my Postgresql (on AWS) and I need to decide on how to store the Social Media/Digital trails from wearables (unstructured and structured) information. I was thinking DynamoDB.

I was also thinking to integrate both databases under Amazon Redshift and to do the analytics (using RapidMinder) from there..... Does it all make sense? Do I really need a data warehouse for this? Will it be more sensible to use just a single DB (Postgresql or Dynamo) for all this without data warehousing? to I am talking about up to 100K records more or less (for the training).... Future data will in the millions.

I get so many conflicting answers and I hope and will appreciate your kindness and advice. Thank you so much in advance!!!


texts: What is the interest to implement a statistics model in a groupby object instead of use the whole groups in a sample?

texts:
I saw this topic and also this one in dask documentation.

I wonder what is the real interest to make a model for each group instead of considering a whole sample where all groups would appear? As far as I know the more informations you have, sure the more complexity you have (not good for overfitting), the more the trained model is fitting the reality. So what is the goal?
More precisely, after you answer the generality above, is it interesting to make odds ratio with a logistic regression where each model is applied on one group (excluding others in each model)? What are the pros, cons, then the limits?


texts: How to define tidy data if there is repeated measures?

texts:
Some time ago I read R for Data Sciece and there is the following definition of tidy data:

There are three interrelated rules which make a dataset tidy:
Each variable must have its own column.
Each observation must have its own row.
Each value must have its own cell.

Back then, the idea seemed quite reasonable but now the concept seems not that consistent to me. In chapter 12 it is said that table1 is tidy and this is how it looks like:

I would expect the country to be the observation unit and therefore, countries must not appear in multiple rows. I expect the following to be the tidy data table of table1:

The problem with my solution seems to be that we have values (of the year) as part of variable names. On the other hand, the dataset table1as it is suggested to be tidy has no obvious observation unit in my understanding. We could say that the combination of the two columns country and yearforms the observation unit but there is nowhere a definition of such a rather complex observation unit (not in the book and not in the publication on tidy data).


texts: Finding user similarities within informal data sets

texts:
I'm new to all this and am putting together a learning project. I've decided on finding similarities between users in a data set such as http://en.wikipedia.org/wiki/Enron_Corpus. After doing a bit of research, I also came across Dataset for Named Entity Recognition on Informal Text. So I'm not short of data or a goal, I need to understand high-level techniques to get there.

One valuable comment noted that this question appears too broad. What I was hoping to find with this question was the breadth of techniques I should focus research on, not answers that are immediately implementable. Please consider vague answers as entirely appropriate!!

Expanding on the goal, I am hoping to discover which authors might have affinity toward each other, or conversely do not care much for each other. So I will definitely need to start with Named Entity Recognition and build a means to organize the documents against those entities. Beyond that, I am not so sure.

What high level concepts should I be looking at? Thanks!


texts: Parsimony principles in the Box and Jenkins approach

texts: What are the parsimony principles in the Box and Jenkins approach? In my understanding it is about parsimony vs overfitting, i.e. how can we best fit to the data that is representative of the data generating process.


texts:
I am building a neural network that consists of an LSTM, dense and dropout layers using Keras to forecast 8 continuous values in the future. I unfortunately have a very small data-set of 56 observations. I divided my data-set into 48 observations for training and 8 observations for testing. I ran my code 5 times in order to split my data-set differently each time. Within each run, I also fit the model 100 times and use the maximum test R-squared to pick the best performing model in each run.
Therefore, I have the following r-squared values: 0.89, 0.90, 0.93, 0.91, 0.87.
If I want to report an r-squared value, which one should I use? How to be sure that I reached the global optimum r-squared value? Also, would the RMSE benefit me in taking such decision?


texts: Confusion with Notation in the Book on Deep Learning by Ian Goodfellow et al

texts:
In chapter 6.1 on 'Example: Learning XOR', the bottom of page 168 mentions:

The activation function $g$ is typically chosen to be a function that
is applied element-wise, with $h_i = g(x^TW_{:,i}+c_i).$

Then we see equation 6.3 is defined as (assuming g as ReLU):

We can now specify our complete network as
$f(x; W,c,w,b) = w^T$
max$\{0, W^Tx + c\} + b$

Wondering why the book uses $W^Tx$ in equation 6.3, while I expect it to be $x^TW$. Unlike XOR example in the book where $W$ is a $2\times2$ square matrix, we may have non-square $W$ as well, and in such cases, $x^TW$ is not same as $W^Tx$.
Please help me understand, if I'm missing something here.


texts: Predicting positive/negative experience with very few labels and labels from only one class

texts:
I have video viewing data (length of session, nb of videos, etc), as well as if the user clicked on the like button. We can use the like button as a confirmation that the user had a positive viewing experience, however, only 0.1% of users click on this button.
I would like to find a way to find users that have a similar data to those who liked the video without having them explicitly click the like button.

I thought about having the like variable be the response variable in a binary classification problem, however, not liking the video does not mean negative experience.

I also thought of maybe treating it as an unsupervised task, where I look if the liked sessions fall naturally inside a specific cluster.

Edit:  I did not make it clear, but the service is similar to Youtube, where we are trying to figure out if a user had a positive viewing experience after clicking on a video. Right now, there is no recommendation engine and this is the first part in building one.

Edit: After the answers, I am leaning more towards approaching this task as an unsupervised learning task, rather than supervised.

Any thoughts how to approach this problem?
Thanks


texts: Difference Bagging and Bootstrap aggregating

texts:
Bootstrap belongs to Efron. Tibshirani wrote a book about that in reference to Efron.

Bootstrap process for estimating the standard error of statistic s(x). B bootstrap sample are generatied from original data. Finally the standard deviation of the values s(x1),s(x2)..s(xB) is our estimate of the standard error of s(x). The bootstrap estimate of standard error is the standard deviation of bootstrap replications. Typical value for B, number of bootstrap samples range from 50 to 200 for stand.error estimation
Breiman wrote in his paper 1996,

What I understand is, that only when it is:

replicate
and random

thats called bagging.
Does it mean, when you just/only do bootstrap without an aggregation at the end you only use the standard deviation of the bootstraps replications togheter, than you call it bootstrap.
If you do the same, but at the end you take an aggregation of voting or anything else, than you call it bagging.
am I right?
So Breiman dont gives an exactly answer of the aggregation method in the end. But his method (voting) is different of the original bootstrap method (standard deviation) in Tibshiranis book which refers to Efron´s original (Efron´s method).
Can we say that this is a different of bagging and bootstrap aggregating?
Of course roughly said that´s the same. (But im looking for the correct answer).


texts: Why the number of neurons or convolutions chosen equal powers of two?

texts:
In the overwhelming number of works devoted to the neural networks, the authors suggest arhitechure in which each layer is a numbers of neurons is power of 2

what are the theoretical reasons(prerequisite) for this choice?


texts: During a regression task, I am getting low R^2 values, but elementwise difference between test set and prediction values is huge

texts:
I am doing a random forest regression on my dataset (which has abut 15 input features and 1 target feature). I am getting a decently low R^2 of <1 for both the train and test sets (please do let me know if <1 is not a good-enough R^2 score).

import pandas as pd
import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.model_selection import train_test_split

# load dataset
df = pd.read_csv('Dataset.csv')

# split into input (X) and output (Y) variables
X = df.drop(['ID_COLUMN', 'TARGET_COLUMN'], axis=1)
Y = df.TARGET_COLUMN

# Split the data into 67% for training and 33% for testing
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.33)

# Fitting the regression model to the dataset
regressor = RandomForestRegressor(n_estimators = 100, random_state = 50)
regressor.fit(X_train, Y_train.ravel()) # Using ravel() to avoid getting 'DataConversionWarning' warning message

print("Predicting Values:")
y_pred = regressor.predict(X_test)

print("Getting Model Performance...")

# Get regression scores
print("R^2 train = ", regressor.score(X_train, Y_train))
print("R^2 test = ", regressor.score(X_test, Y_test))

This outputs the following:

Predicting Values:
Getting Model Performance...
R^2 train =  0.9791000275450427
R^2 test = 0.8577464692386905

Then, I checked the difference between the actual target column values in the test dataset versus the predicted values, like so:

diff = []
for i in range(len(y_pred)):
    if Y_test.values[i]!=0: # a few values were 0 which was causing the corresponding diff value to become inf
        diff.append(100*np.abs(y_pred[i]-Y_test.values[i])/Y_test.values[i]) # element-wise percentage error

I found that the majority of the element-wise differences were between 40-60% and their mean was almost 50%!

np.mean(diff)
>>> 49.07580695857447

So, which one is correct? Is the regression score correct and my model is good for this data, or is the element-wise error I calculated correct and the model didn't do well for this data? If its the latter, please advise on how to increase the prediction accuracy.

I also checked the rmse score:

import math
rmse = math.sqrt(np.mean((np.array(Y_test) - y_pred)**2))
rmse
>>> 3.67328471827293

This seems quite high for the model to have done a good job, but please correct me if I'm wrong.

And I also checked the R^2 scores for different number of estimators:

import matplotlib.pyplot as plt
model = RandomForestRegressor(n_jobs=-1)
# Try different numbers of n_estimators
estimators = np.arange(10, 200, 10)
scores = []
for n in estimators:
    model.set_params(n_estimators=n)
    model.fit(X_train, Y_train)
    scores.append(model.score(X_test, Y_test))
plt.title("Effect of n_estimators")
plt.xlabel("n_estimator")
plt.ylabel("score")
plt.plot(estimators, scores)

Please advise.

I tried using linear regression first, and got a very high MSE (which is why I was trying out random forest):

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

lr = LinearRegression()
lr.fit(X_train, y_train)
y_pred = lr.predict(X_test)

# The coefficients
print('Coefficients: \n', lr.coef_)
# The mean squared error
print("Mean squared error: %.2f" % mean_squared_error(y_test, y_pred))
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % r2_score(y_test, y_pred))

Coefficients:
 [ 1.93829229e-01 -4.68738825e-01  2.01635420e-01  6.35902010e-01
  6.57354434e-03  5.13180293e-03  2.84015810e-01 -1.31469084e-06
  1.95335035e+00]
Mean squared error: 86.92
Variance score: 0.08


texts: Which classification algorithm to choose for classifying driving patterns (GPS coordinates) and mapping them to drivers?

texts:
I have been given 6 data sets out of which 5 are training sets. The 5 data sets correspond to 5 drivers.

Each row of a data set consists of a time-stamp followed by a poly-line. The poly-line gives the GPS coordinates (latitude,longitude) of the driver taken every 15th second. So the length of this line gives the time duration of each trip. I have to classify the poly-lines corresponding to each time-stamp to one of these 5 drivers in the last (test) data set.

I need assistance in choosing a classification algorithm. I have only implemented Naive Bayes (text classification) before but I don't think it will work here. I'm having trouble visualizing the approach to solve this problem.


texts: How to shuffle input data using stochastic gradient decent?

texts:
Let's say that I have input with size $n\times n$ and after that I operated shuffled operations and get the output with the same dimension. But I don't know if we can do that or not. The idea is to permute all the configuration and find the minimum configuration when given loss. But the operation became very expensive with the growth of $O((n^2)!)$, and with each iteration we need to compute loss separately. So is there an optimization for such problem or it just not feasible?

In other words, is there a gradient correlation between different configuration that will lead to the local minima of the loss function?


texts: Appropriate Approaches for Unsupervised Textual Classification

texts:
Let's say I take 1000 dissertations from a variety of different fields, and thereafter extract the top 100 words from each dissertation. What would be the best approach to categorize them into STEM/non-STEM without having a sample of STEM/non-STEM dissertations to compare? Is there an R package that will sort them based on shared language without pre-tagging words (so a bit different from semantic or sentiment analysis)?
Presumably they will all share a core vocabulary, and there will be some overlap here and there, but presuming we can know from the top 100 words, what tools would assist in that? For this type of work, I've seen far more examples of supervised learning, but I'm much more interested in seeing how they could possibly be grouped, even if the end result isn't a division into majors per se.
The setup/fact that they're dissertations/theses doesn't matter. A similar idea could be to take two playwrights and analyze their plays to see if they're correctly sorted by authorship. What approach should I be taking here?


texts: Poisson distribution, Standard deviation, fitting line

texts:
Let's say I have system A, B, C, and D. Each system contains 10,000 numbers generated by Poisson distribution. The difference is the mean is different for different systems. I calculated the std dev for each distribution corresponding to each system. I plot the standard deviation with respect to mean numbers. (I observe std deviation increases with mean). I want to fit the plot with some line, which gives me the general estimate of std dev with respect to the mean number. I need help in finding the fitting line!
Thank you!


texts: Is it faster and better to train a GAN on just one digit as opposed to the whole mnist dataset?

texts:
When going through an introductory GAN tutorial to generate mnist like handwritten digits I wondered whether the systematic variance in the training data due to the different digits makes the model harder to train.

Would't it be easier to train a model if all real samples would be 1s instead of 0-9s?

My question

Would an approach where I train a separate model to generate each digit (0-9) separately using only training data of one type of digit be feasible at all?
Would training those models for just one digit be faster (i.e. need less epochs to reach a certain accuracy/quality) than a model using all of mnist.

Edit

I found an acceptable answer for me (see below) but obviously I am happy for anyone else to chime in.


texts: Multi task learning architecture for Multi-label classification

texts:
I am working a classification problem. The dataset was collected from  Painters by number, a competition hosted by kaggle. The task is to identify painter,style and genre given paintings.

So far, I trained individual models to predict painter,style,genre given paintings. Now i would like to incorporate Multi task learning (i.e) Developing a single model which can predict all three tasks.

The issue i am currently facing is in designing architecture.

      Model                         No of classes (Softmax)
   (Individual Models)
   --------------------             -----------------------
 Model predicts painter                     8 (8 - painters)
 given paintings

 Model predicts style                       10 (10 style classes)
 given paintings

 Model predicts genre                       23 (23 genre classes)
 given paintings

I don't know how to combine the above models in keras. Any suggestions or feedback would be helpful.


texts: What are the differences between Apache Spark and Apache Flink?

texts:
Both Apache-Spark and Apache-Flink projects claim pretty much similar capabilities.

what is the difference between these projects. Is there any advantage in either Spark or Flink?

Thanks


texts: How can I predict rank of a team based on list of team members and past placement?

texts:
I just started with ML, so this could potentially be a pretty stupid question, plz forgive me.
Here's the gist of the problem:
I have a list in json format like this:
[
  {
    // Final rank of the team after all the matches are done
    "rank": 7,
    // List of team members
    "characters": [
      { "char_id": "my_char_1", "level": 2, "item_ids": [1, 2, 3] }
      ...
    ]
  }
]

I want to predict the rank for a given "team" of characters without looking at the competitors.
Basically, I want to go from this:
[
  { "char_id": "my_char_1", "level": 2, "item_ids": [1, 2, 3]  }
  ...
]

to a single output neuron with a value from 1 to 100.
I parsed the source data into the first json structure, but I'm currently stuck at trying to give Tensorflow pairs of data (team + past rank), ideally in the form of two arrays where elements at a given index belong together.
TLDR:
Assuming this is possible at all (and not a lack of information or something), how do I feed Tensorflow this example data (list of team members -> past rank) properly to predict the future rank of a completely new team?

I've found this question which is somewhat similar:
Predicting outcome of MOBA team games
Based on this answer, I'm assuming I'll have to normalize the data, so instead of an array I would need a map like this:
{
  "my_char_1": null,
  "my_char_2": {
    "level": 2,
    "item_ids": {
      "1": true,
      "2": false,
      "3": true
      ...
    }
  },
  ...
}

However, I struggle with the act of passing the data itself, not so much how to prepare the json structure. Basically, take whatever json structure would be optimal and assume it already exists.


texts:
How do we design a CNN for ordinal classification?
I am trying to analyze plant leaf images for diseases. I've got the disease type classification working. Now we need to classify the magnitude of the disease affliction on a scale of 1 to 10 (ordinal scale), where 1 is almost no trace of diseased parts on the leaf and 10 is completely diseased.

Is it possible to construct a CNN for this task?

How do I deal with the imbalance in training samples for each ordinal level?


texts: Using nearest neighbour as mapper of xy coordinates

texts: This is my first post here so I apologize if this is not right place for this kind of question. I am looking for some tips on using (k)nearest neighbor algorithm as a mapper of hypothetical position of an entity in 2d. space. Training data set contains dimensions (W and H) of space the entities are in, entity type, and entity position (X and Y); for example: W: 10, H: 5, type: someType, X: 7, Y: 3. Different types are constrained to occupy only certain regions of space, eg. someType's X position can be only in range of 0-7, Y in range 0-3, and someType2's X can only be in range of 7-10, Y in range 3-5. My question is how to express an algorithm that based on provided training data (which maps types to correct positions in given space) outputs aproximate position. To be more clear: with input W: 15, H: 7.5, type: someType algorithm should output X: someX, Y: someY. Since same types are clustered I thought that nearest neighbor is right approach, but the problem is not exactly to provide some single label or classification but rather output some compound value X, Y.


texts:
I am involved in twitter analysis data. I want to find trending topics in tweets  with some hashtags, like #finance or #technology. I have a hugh data set of tweets and now I need to analyze them.

I need to recognize topics, if there are. They way I'm approaching this is, first, performing a vector representation of each tweet, with a tfidf technique, and then, build groups of them based on their cosine similarity.

Are there common techniques in tweets analysis?


texts: Setting up Neural Network for this problem

texts:
I have a question regarding neural networks considering I am not an expert in NN.
Assume have a 5 by 5 grid that depending on me pushing any square (or combination of squares) some of those squares (not necessarily the squares I have pushed) will light up. My question is:

Can we set this problem as a NN problem if I have a set of input and outputs? Assume input layer with 25 neurons where all are zero except the ones are pushed and output layer is another 25 neuron where all are zero except the squares that light up. Assume in my training data set I have done this experiment 50 times.
Let’s say, in my training set square (3,3) has never been pushed. Can my trained model predict what would the response be if the square is pushed. So basically if a square has not been pushed in training set can NN still be used for prediction?
At the end, if all this is possible how would you set up a problem like this? Which NN method would you choose?


texts: Using pretrained LSTM and Bert Models in CPU Only Environment - How to speed up Predictions?

texts:
I have trained two text classification models using GPU on Azure. The models are the following

Bert (ktrain)
Lstm Word2Vec (tensorflow)

Exaples of the code can be found here: nlp
I saved the models into files (.h5) for later use. The files are big e.g. 27,613kb for the lstm and 1.2 gb for bert.
I loaded the models and in a computer where only CPU is available. They both work fine but the model.predict(text) function is super slow predicting the class of the text e.g. on average 1 tweet sized message per second.
Adding GPU on the computer is not an option. I wonder if there is another way to make it run faster? e.g. train the models in a different way (without compromising accuracy) or save the model in a different file format?


texts: Deep learning theory: why are hidden layers necessary?

texts:
For this question, I’ll refer to the popular YouTube video by 3Blue1Brown on deep learning applied to recognition of written numbers.
The video describes a neural network with the following layers:

Input (individual grayscale pixels)

Small-scale feature determination (e.g., the top quarter of the loop in the number 9)

Larger-scale features (e.g., the entire loop in the number 9)

The output layer, which shows the probability of an input image being each of the numbers 0-9

I’m also going to try to read through the entire wiki section here, and I’m currently on the neural networks page.
I particularly like the explanation of coefficients... “A node combines input from the data with a set of coefficients, or weights, that either amplify or dampen that input, thereby assigning significance to inputs with regard to the task the algorithm is trying to learn; e.g. which input is most helpful is classifying data without error?“
Essentially, it’s saying that each input has some level of importance to each output, which to me begs the question... are component features/hidden layers necessary at all?  In the handwriting example, couldn’t every input node be connected to every output node without the use of hidden layers?  The idea is that all of the high-weight input pixels or a given output would still have high weights for that output, but the network would skip the feature/aggregation stages.  Is this just a matter of training efficiency (i.e., to prevent duplication by effectively extracting the same features more than once)?
Also, do the connections between various nodes need to specifically chosen so that the number of nodes and number and selection of connections are intelligently chosen?
Is it accurate to say that a sufficiently deep neural network looks to find the significance of all relevant combinations of input values and that’s basically all it’s doing?


texts: Balanced Linear SVM wins every class except One vs All

texts:
I am training a normal and a balanced Linear SVM using imbalance data, and testing both using F1-score.

(By balanced Linear SVM, I mean that each observation has a weight inverse to its frequency, so that it "oversamples" from the minority class and "undersamples" from the majority class.)

Not surprisingly, using a multi-class dataset and solving it individually as one-vs-rest binary problems, balanced Linear SVM beats unbalanced Linear SVM on every class:

                   normal vs balanced
target 01 - scores: 0.272 vs 0.608
target 02 - scores: 0.391 vs 0.587
target 03 - scores: 0.433 vs 0.546
target 04 - scores: 0.659 vs 0.655
target 05 - scores: 0.000 vs 0.257
target 06 - scores: 0.431 vs 0.475
target 07 - scores: 0.000 vs 0.249
target 08 - scores: 0.000 vs 0.053
target 09 - scores: 0.576 vs 0.155
target 10 - scores: 0.000 vs 0.550
OneVsRest - scores: 0.565 vs 0.540

But when it comes to using a OneVsRest classifier (last row), the unbalanced Linear SVM beats the Linear SVM, using the average of the F1 scores. This happens to me in a lot of datasets.

Here is the code I have used:

# -*- coding: utf-8 -*-

from sklearn.svm import LinearSVC
from sklearn.multiclass import OneVsRestClassifier
from sklearn.cross_validation import StratifiedShuffleSplit
from sklearn.metrics import f1_score
import numpy as np

d = np.loadtxt('yeast.csv', delimiter=',')
X = d[:, 0:-1]
y = d[:, -1]

print '                     none vs balanced'
for target in np.unique(y):
    scores = [0] * 2
    k = 6
    for tr, ts in StratifiedShuffleSplit(y, k, 0.2):
        for i, w in enumerate([None, 'balanced']):
            _y = (y == target).astype(int)
            yp = LinearSVC(class_weight=w).fit(X[tr], _y[tr]).predict(X[ts])
            scores[i] += f1_score(_y[ts], yp) / k
    print 'target %02d - scores: %.3f vs %.3f' % (target, scores[0], scores[1])

scores = [0] * 2
for tr, ts in StratifiedShuffleSplit(y, k, 0.2):
    for i, w in enumerate([None, 'balanced']):
        m = OneVsRestClassifier(LinearSVC(class_weight=w))
        yp = m.fit(X[tr], y[tr]).predict(X[ts])
        scores[i] += f1_score(y[ts], yp, pos_label=None, average='weighted') / k
print 'OneVsRest - scores: %.3f vs %.3f' % (scores[0], scores[1])

As the dataset, I have used here Yeast (UCI), for which a sklearn-ready version can be found here.

Results:

                     none vs balanced
target 01 - scores: 0.241 vs 0.612
target 02 - scores: 0.402 vs 0.604
target 03 - scores: 0.444 vs 0.552
target 04 - scores: 0.666 vs 0.650
target 05 - scores: 0.000 vs 0.286
target 06 - scores: 0.370 vs 0.500
target 07 - scores: 0.000 vs 0.280
target 08 - scores: 0.000 vs 0.082
target 09 - scores: 0.563 vs 0.147
target 10 - scores: 0.000 vs 0.511
OneVsRest - scores: 0.567 vs 0.543

Isn't this surprisingly? I know that OneVsRest has a lot of ties, and so scores will be used, as determined by the distance to the separation hyperplane. Still, why does balanced Linear SVM loses the war when it keeps wining every battle?

EDIT: I think the fact that one model wins pretty much every "battle", but not the "war" is related to the fact that One Vs Rest uses confidence scores, and not the actual classification, which makes sense since it avoids ties, and when score[k1] > score[k2] we can assume it prefers k1 > k2. See wikipedia.


texts: How to build an encoder using a distance matrix

texts:
I have a similarity/distance matrix:

   a | b | c
 a 0 | 1 | 2
 b 1 | 0 | 3
 c 2 | 3 | 0

I want to build an encoder/model that learns an n-dimensional representation of each of the points in the dataset s.t. the euclidean-difference between the representations produces the difference provided in the matrix, e.g. distance(a,b) = 1 etc.

Any ideas?


texts:
How can I concatenate matrices with unequal rows by intercalating the rows? For example, I want to concatenate A matrix ( dim(A) = 15 20 ) and B matrix (dim(B) = 17 20) and I want the result to be a matrix of 32 rows by 20 columns. I want result matrix with one row from matrix A and the second row from matrix B. Simple rbind just stacks up one matrix on top of other.

The following function works in case of equal number of rows:

# drop false ensures that coercion does not change the data structure.
interleave(A, B, drop=FALSE)

But I want the rows to be interleaved and then the remaining rows from B matrix to be added at the end. If the number of rows from one matrix is added and no more are remaining then the remaining rows should get rowbind (normally stacked).

Is there any way that I can tweak this interleave function from gdata package in R? or is there any other way?


texts: Data Preprocessing, how separate background from image to detect animals?

texts:
Hello I'm looking for some advice making a animal Classifier from trap cams, I applied SIFT to detect points of interested but it turns out that most of the points were in the background and less in the animals. My intention is to recognize animals from a dataset, any ideas that you can bring me to achieve what I'm trying to do?

Images are something like this and the keypoints :

Any other options or suggestions? to change the way I'm going now.


texts:
I am doing a linear regression on log-transformed data and I use the bayesian approach to model the predictive distribution and construct my 90% prediction Interval.
The problem with this approach is that my confidence interval are very large because of the exponential transformation. for an aleatoric standard deviation of 5% on my data set, the relative prediction interval width is of the order of 1.5 which is very large.
I think that the exponential transformation of the irreducible noise is making my confidence interval very large and not fitted to the data and I was wondering if there is any other approaches to attack this problem instead of the bayesian approach on the log-transformed data:
Any solutions to obtain smaller prediction intervals ?


texts: Daily new data for my neural network, and I want transfer(?) learning

texts:
I made my neural network, it is pre-trained for 180 days of data.

It filters the fraud data of credit cards everyday and 1-days new data is comming in.

And I also want after the filtering,

  I want to re-train my AI-model but I just want to use new 1-day
  data only(because training neural network is really time-consuming work).

My AI model is 0(not fraud)/1(fraud) classification model. I want to change my neural net by 1/181.... because the amount of data is just a one day

How I should train the neural network? If I use just 1 day data and run many epoch(time), It will over-fitted... By early stopping, It seems the one-day data train is not.... sufficient......

I think memory like LSTM may need my neural net... What design of neural net is best for my situation??


texts: Is it true that a neural network with many layers converges to a solution?

texts:
For one hidden layer exist Universal approximation theorem by  George Cybenko.

But for multi layers net converges result is stohastic or determ and guaranteed as one hidden layer net?
And is there an appropriate proof?


texts:
I have two csv files,and if I do:

df_compiler = pd.read_csv('sub_compiler.csv')

I obtain:

and for the other csv file I do:

df_opt = pd.read_csv('sub_opt.csv')

I obtain:

now what I would like to do is merge these two csv files and obtain something like:

can somebody please help me doing this? Thank's in advance.


texts: Classification model for recommender system?

texts:
I have some data for various customers choosing one of 'n' products or no product. I have some useful features for each customer. I can build a multi-class classification problem out of this data and use a classification model (say, random forest) to learn the data. This model will then output one of the following categories:

[0, 1, 2, ..., n]

where 1, 2, ..., n is the 1st, 2nd etc. products, and 0 is when the customer chose not to buy anything. I want to make this model as a recommender system, i.e., when a new customer (along with all other features) comes along, I know which one of the n products he will more likely buy.

The question is this: in the above setting, I don't know what I should do when the models outputs '0' for the new customer. What should be my action when such a situation, i.e., the new customer is not likely buy any of the products? Or should I formulate the classification problem in some other way? Is there any way we shouldn't formulate the problem as a classification problem at all?


texts:
As far as I know, to evaluate synthesized images it is proposed to use:

human scoring,
"Inception score",

where in the second case the quality is rated based on a pre-trained Inception model’s ability to classify them.
I find "Inception score" method rather complex, but that's not the point. Is there any other method to evaluate synthesized images? For example, calculating RMSE of the result and ground truth images, or something like that.


texts:
I was wondering, is it a proper method to convey information via separate Recall and Precision Confusion matrix? I recently came across a paper which reported the following scores. I am puzzled and unable to interpret them. Is this a usual standard in the academics?

Edit 1: The paper is related to activity recognition.


texts:
I want to train a model to approximate the probability of an event between two objects based on previous occurrences. My input training data are the parameters (six in total) of the pairs that have taken part in a past event and the output training data are the computed probability of the event. So one case would be:

X_train:

param1_obj1,param2_obj2
param2_obj1,param2_obj2
..., ...
param6_obj1,param6_obj2

Y_train:

probability of event

My idea is to try to train a neural network and compare it with other models

My question is: How should I format the inputs to feed them to the network properly? Would feeding the (6,2) array work?


texts: Error when comparing testset with training set in ARIMA R

texts:
I am working on a monthly dataset (timeseries) using R. The dataset consists of 159 rows starts from Feb 1996 to March 2004. I set aside the last 12 months of data (Apr 2003- March 2004).
Performed ARIMA modelling and got the model.
When I try perform validation like:
pred <- Arima(test,model=train_model)
It throws me an error:
Error in stats::arima(x = x, order = order, seasonal = seasonal, include.mean = include.mean,  :
  too few non-missing observations

I'm not exactly sure what is causing this error.


texts: If I have an ROC for a single classifier [y(x) in the range 0...1] that is 'worse than random', namely the AUC of the ROC is less than 0.5,  would a classifier that reversed the class predictions [y'(x)=1-y(x)] be better than random to the same degree?


texts: Why not use max(returns) instead of average(returns) in off-policy Monte Carlo control?

texts:
As I understand it, in reinforcement learning, off-policy Monte Carlo control is when the state-action value function $Q(s,a)$ is estimated as a weighted average of the observed returns.

However, in Q-learning the value of $Q(s, a)$ is estimated as the maximum expected return.

Why is this not used in Monte Carlo control?

Suppose I have a simple 2-dimensional bridge game, where the objective is to get from a to b. I can move left, right, up or down.
Lets say a reward of +1 is given for reaching b and -1 otherwise.

a|x|x|x|x|b

If the agent reach the final x (using epsilon-greedy policy for instance), then goes up to get a reward of -1. Then in the next episode, I reach the final x and go right to get a reward of +1.

Why wouldn't I update all the steps leading to 'b' to +1, as I would do in Q-learning?

I'm assuming the environment is deterministic in this case, so I don't overestimate $Q(s,a)$ based on an unlikely event.


texts: Trying to make compelling plot for classification results with python

texts:
I've been working on a classification problem and have some good results, but now I struggle with trying to put together a good plot to illustrate the probabilities for each prediction.  Here is my current data:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

df_voting_output.head(n=5)
    prob    actual  pred    correct
0   0.460200    0   0   1
1   0.548478    1   1   1
2   0.270609    0   0   1
3   0.686557    0   1   0
4   0.527935    0   1   0
5   0.098687    0   0   1

I've been able to create a bar chart with probabilites using this code:
plt.bar(np.arange(len(voting_predictions[:,1])), voting_predictions[:,1])
plt.xlabel("record number")
plt.ylabel("probability")
plt.title("Classification Probabilities")
plt.show()

However, I'm thinking there has to be a better way to include more information and a key.  I'd like the "correct" labels to be clear to see, so from a visual standpoint you can see how the probabilities relate to a correct classification.


texts: References/tutorials about data mining and machine learning

texts:
I am learning data analytics and I wonder if there are some good references and tutorials about machine learning, data analytics and data mining?
What I'm searching for is an understandable reference/tutorial, which isn't very technical and isn't very basic either, in other words the material begins with the basic steps towards advanced steps.
Thank you.


texts: How to tune the hyperparameters in graphical models such as a Markov random field?

texts:
What's a good way to tune the hyperparemter of a Markov random field (MRF) with a Gaussian prior that's not uncommon in image segmentation tasks? The structure of the graphical model is not a big issue, but we can assume a 4-connected graph. Say there's no ground truth for training, but we only use the MRF for inference task in segmentation.
It seems like for graphical models, people tend to set the values of hyperparameters without tuning it (at least approximately).


texts: Algorithm for backpropagation through time

texts:
I am reading through this article trying to understand the bptt algorithm, in the context of an RNN. However there is one part I don’t understand:

layer_1_delta = (future_layer_1_delta.dot(synapse_h.T) +
    layer_2_delta.dot(synapse_1.T)) +
    sigmoid_output_to_derivative(layer_1)

# Located on line 99 in the article linked above

I can’t seem to figure out which mathematical formula this links to in the bptt algorithm to understand what it does, so could someone please provide it? Many thanks


texts:
For example, I have some time series. How can I change my data, so it will not be obvious to understand what was original values?

Ideally transformation would allow to revert and reconstruct original data with some noise(or to save the relationship between time series values with no option to revert to original values)

I thought of: multiplying each value of series by some number, which will do shift and scale operation or I can compute some continuous function(like log), where argument is my time series data.

Nevertheless, I believe there should be more advanced method.
Thanks!

Update:
After careful considerations, and answer of @D.W. below and trying to transform data, I found that I spend more time on this than on main problem I have.

Initial problem was twofold:

1) to send real data to home computer to play at home any time I want.

2) to send real data to home computer, transform it a bit, to share with people in the web, to get some advise on my main problem.

After thinking, stressing, askying, googling I decided to drop the idea to do anything.
Few good links to people who have the same problem and are intended to transfer, transform data:

For problem 1) I believe stream cipher is a an option to read more about ( as it transforms your data to a random string). Result is bad, as if one will look at your working computer logs, you are busted.

For problem 2) read about laplace noise. Also, believe to be bad.

Regards.


texts:
I have an extremely sparse user items ratings matrix with 0.018 % non NA values. Correct me if I am wrong but I think we need a lot of products compared to number of users to build a recommender system. I have around 20,000 users and 50,000 products.

Is is possible to build a recommender system with this dataset? If so, what would be the most appropriate approach?


texts: can i scale features within each input catgory differently

texts: i am trying to predict laptop prices from previous sales. based on a text description of the laptop condition and the sale price. the affect of the condition on the final price is going to be different for a high value laptop and low value laptop. e.g. a laptop with a broken screen may be worth 25% of a laptop with in excellent condtion. can i feature scale each laptop model based on the range of sale value for the model and then try and predict the scaled value from the text description in my test set. and then transform back to get the predicted price. or is there a better way to capture the fact that the affect of a condition variable will be different for depending on some baseline value for the model.


texts:
I tried to standardize the training data with samples of 629,145 rows and 24 features:

from sklearn import datasets
import pandas as pd
from sklearn import svm
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import StandardScaler

df = pd.read_csv('mydata.csv', dtype='object')

#manually choosing 24 features
X=df.loc[:, ['Bwd Pkt Len Min','Subflow Fwd Byts','TotLen Fwd Pkts','TotLen Fwd Pkts','Bwd Pkt Len Std','Flow IAT Min',
             'Fwd IAT Min','Flow IAT Mean','Flow Duration','Flow IAT Std','Active Min','Active Mean','Fwd IAT Min',
             'Bwd IAT Mean','Fwd IAT Mean','Init Fwd Win Byts','ACK Flag Cnt','Fwd PSH Flags','SYN Flag Cnt','Fwd Pkts/s',
             'Bwd Pkts/s','Init Bwd Win Byts','PSH Flag Cnt','Pkt Size Avg']]
Y= df['Label']

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.4,random_state=42) # 60% training and 40% test
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

#Create a svm Classifier
clf = svm.SVC(kernel='rbf') # not linear Kernel
clf.fit(X_train, y_train)

Already 6 hours and SVM does not converge. Same data is converged with other algorithms very fast as RF and I know it is quite normal as SVM considered computationally high algorithm compared to KNN and RF.. I read quite a bit of questions/answers and articles.

I wonder how can I track and analyze the problem visually (probably plotting some graphs as cost function or ?).
Can parameter tuning (C parameter) be of the help and speed up this?
What could be your advice?

Thanks very much


texts: How to have Multiple labels in a single video?

texts:
I am building a Tennis stroke classification system using CNN.
I assume each stroke contains 3 steps/classes ('Ready', 'Impact', 'Finish'). I want to train a model which will predict whether the input video contains these steps/classes in it.
I have tried training 3 models for each step as binary classification.
Example of one step model classes:
1 - ready
0 - not-ready(other incorrect steps).

But this method failed since there are more features in 'not-ready' class. I got only 4% accuracy.
Can anyone help me to find a solution for this problem.


texts: How do I get the values of y_true and y_pred in the confusion matrix?

texts: I am using the weka program and i want to know the values of y_true and y_pred through confusion matrix that appeared in the classification results , because I want to calculate the balanced multiclass accuracy value .


texts: Should I apply 1D or 2D CNN on binary text classification?

texts:
I am trying to train a text classification model. For all sentence examples, I limit them up to 32 words, and if there are not exist 32 words, I am creating zero pad arrays. To convert each word to vector, I used a pre-trained word2vec model.

In the final setting, the shape of my data is :

x_train: 15000 samples and each sample has 32 vectors in which each vector size is 100. (15000, 32, 100).

y_train: 15000 binary targets (15000, 1)

So my question is that, should I apply 1D CNN on my x_train or 2D CNN? I think that I can do it in both ways, but is there the main correct approach to this kind of problem? I read some stuff about 1D CNN on text classification but also there are some examples with 2D CNN. What are the cons and pros?


texts: Link Prediction based Similarity Indices

texts: So, I was reading some Link Prediction based algorithms and similarity indices. I came across two random walk based indices - Local Random Walk(LRW) and Superposed Random Walk(SRW). I read the standard and original papers on both, but couldn't understand them. Can someone please help me with understanding both of them properly?


texts: Is there a good online course for working with sklearn MLPClassifier?

texts:
I'am implementing an App for which I need a neural network. Because: I want to classify each DOM-text-element of an webpage which conains any curriculum vitae of a person. The neural network should be able to classify each DOM-text-element then if it is e.g. the name of the person, the birthday of the person, the last studies of the person, ... or only noise.

I want to use for this issue the sklearn MLPClassifier in python. But I neeed much more knowledge in:

a) which data should the feature vector contain (CSS properties, ...))

b) how many data  I need

c) which parameters needs the MLPClassfier for the best result (or only at least working)

Does anybody know a good online course where I cann get a better understanding of these things?


texts:
I'm using spark with scala to implement majority voting of decision trees and random forest (both are configured in the same way - same depth, the same amount of base classifiers etc.). Dataset is split equally among base classifiers for majority voting. Nemenyi test shows, that majority voting is significantly better (for 11 benchmarking datasets from keel).

From what I understand, the difference between those two methods is that data used to train random forest (base classifiers) might not sum up to the whole dataset. Is my understanding correct? If so, what might be the reason for the observed difference?

Also, could you point me to any articles comparing those two methods?

Edit: If someone was interested in this topic, here's an article comparing bagging with horizontal partitioning in favor of the latter.


texts: What should be the value of parallel iterations in tensorflow RNN implementations?

texts:
tf.nn.dynamic_rnn() and tf.nn.raw_rnn() take in an argument called parallel_iterations. The documentation says:

  parallel_iterations: (Default: 32). The number of iterations to run in parallel. Those operations which do not have any temporal dependency and can be run in parallel, will be. This parameter trades off time for space. Values >> 1 use more memory but take less time, while smaller values use less memory but computations take longer.

If memory is not the constraint, should i not always keep its value equal to the number of processors available with me on GPU.

Here i have my first question - Am i right in thinking that number of processors on GPU mean number of parallel floating point computations it can undertake (i don't suppose GPU's too implement 2 threads per processors as in CPUs)?

Now, GRU machinery is: (please note the biases are not showcased in the image but are to be used for better performance)

,

To me the value 32 seems too low to be kept as default. If i have a GRU with 250 dimensional hidden state and 100 dimensional input i would have (2 x 250 x 350 x batch_size) floating multiplications that i can run in parallel (computing r and u) followed by (250 x batch_size) floating point additions that i can run in parallel followed by (250 x batch_size) sigmoid applications which are at least (250 x batch_size) potential parallel computations. Then it is a (250 x batch_size) parallel computations in application of reset gate followed by (1 x 250 x 350 x batch_size) parallel computations followed by (250 x batch_size) parallel computations for biases. Now in the final step it is (250 x batch_size) floating point computations to calculate (1 - update gate) followed by (500 x batch_size) parallel floating point computations for output state. To me it seems the bottlenecks in parallel implementations using GPU would be presence of too many low sized layers which, for instance here, are bias additions after matrix multiplications (at any rate they are still 250 x batch sized big here so will anyway use GPU to the maximum)

Is the way i am thinking in the second paragraph correct while analyzing a deep network for a vague idea on how much the hardware would be able to impact its running time


texts: Data scaling for time series forecasting

texts: I am doing a time series forecasting project, where I make one step ahead forecasts meaning that my test set is only a single observation. Should I scale the test data as well as my training set and if so do I just lump them together for scaling?


texts: Understanding why my binary classification is approaching 50% accuracy using TensorFlow and Keras

texts:
I'm using the EMBER dataset (malicious and benign binaries).  I used
the built in EMBER code to vectorize the data and then filtered the
unlabeled from the training data.
https://github.com/endgameinc/ember

Now I'm building a very simply NN using TensorFlow and Keras and no
matter what parameters I play with it seems that the accuracy
approaches 50%.  For a comparison the EMBER team get's 98% when
using a Decision Tree (LGBM i think).

as an example this is one of the many things I've tried.

model = keras.Sequential()
model.add(keras.layers.Dense(100, input_dim=ndim, activation='relu'))
model.add(keras.layers.Dropout(0.5))
# model.add(keras.layers.Dense(100, activation='relu'))
# model.add(keras.layers.Dropout(0.5))
model.add(keras.layers.Dense(1, activation='sigmoid'))
model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])
model.fit(X_train,y_train, epochs=10000, batch_size=100)

In this instance I reduced my dataset to 1000 items from 600K to
attempt more epochs.  But either way I seem to approach .50.  Here's
an example epoch result

Epoch 2588/10000 1000/1000 [==============================] - 0s
86us/step - loss: 7.6877 - acc: 0.5180

I feel like I must be missing something fundamental as at some point
the training data should get better even if it's just due to over
fitting.  Also, when the model completes and I evaluate the test
data my accuracy is also almost exactly .50

EDIT 1: How I got the data
The files X_train.dat etc were generated via the code in the EMBER repository, and so should be correct as is.

data_dir = "ember"
#taken from current version of PEFeatureExtractor in Ember
ndim = 2351
#at the moment these are simply just copies of the original
X_train_path = os.path.join(data_dir, "X_train.dat")
y_train_path = os.path.join(data_dir, "y_train.dat")
X_train = np.memmap(X_train_path, dtype=np.float32, mode="readonly", shape=(900000, ndim))
y_train = np.memmap(y_train_path, dtype=np.float32, mode="readonly", shape=900000)

not_negative_one = lambda x: not x == -1
vector_func = np.vectorize(not_negative_one)
condition = vector_func(y_train)
print("The total number of entries should be 900k: and is {}\n after the filtering the total number should be 600k: and is {}".format(len(condition),len(new_condition)))
y_train = y_train[condition]
X_train = X_train[condition]
print("Both y_train and X_train should now have the lengths of 600k: X={} y={}".format(len(y_train),len(X_train)))

data_dir = "."
X_train_path = os.path.join(data_dir, "X_train_no_unlabeled.dat")
y_train_path = os.path.join(data_dir, "y_train_no_unlabeled.dat")
new_X_train = np.memmap(X_train_path, dtype=np.float32, mode="r+", shape=(600000, ndim))
new_y_train = np.memmap(y_train_path, dtype=np.float32, mode="r+", shape=600000)
new_X_train[:] = X_train[:600000]
new_y_train[:] = y_train[:600000]

new_y_train.flush()
new_X_train.flush()

The above code was run once, then the below is run every time

#taken from current version of PEFeatureExtractor in Ember
ndim = 2351
X_train_path = os.path.join(data_dir, "X_train_no_unlabeled.dat")
y_train_path = os.path.join(data_dir, "y_train_no_unlabeled.dat")
X_train = np.memmap(X_train_path, dtype=np.float32, mode="c", shape=(600000, ndim))
y_train = np.memmap(y_train_path, dtype=np.float32, mode="c", shape=600000)

#randomly permute data points
inds = np.random.permutation(X_train.shape[0])
X_train = X_train[inds]
y_train = y_train[inds]
inds = np.random.permutation(X_test.shape[0])
X_test = X_test[inds]
y_test = y_test[inds]

tmp_list_y_train = list(y_train)
print("The training data consists of {} items, with {} malicious and {} benign".format(len(tmp_list_y_train),tmp_list_y_train.count(1),tmp_list_y_train.count(0)))

EDIT 2: I just tried tho model with two digits of the MNIST data set, and the model works just fine.  So the model itself seems to be fine, but there is something wrong with the data set.  I'm uncertain what that could be at this time.


texts: How to use survival analysis for predictive maintenance for time series data?

texts:
So, I have a dataset with daily operating conditions for different machines and a flag saying if it failed or not.
Here is a snapshot of the data.

How can I use survival analysis or any other algorithm to calculate when the machine is expected to fail in the future?
What I understand is that I can use survival package in R, but I am not able to use it for a time series data.


texts:
I have a lot of images and I would like to be able to classify them into two groups: one containing images with watermarks and one containing images without any watermark.

There are about 40 different watermarks. I created "fake" watermarked images to train a CNN and it worked very well on the "fake" validation set but not on the real images. Plus it was a long shot because I would have needed to train a model for each watermark (and I don't have the original watermark) or train a big model.

I quit the watermark approach to try and find text. So I tried OpenCV text detection and it really wasnt working since the text is crooked and not that different from the background.

Is there an easy solution I missed? Any idea is welcolme. I am kinda new to machine learning :)


texts: Create a framework to test effectiveness

texts:
I’m going to make up the use case here. Let’s say I’m testing an equipment with software. I can configure the test software to run for any duration - 2 minutes or 8 hours. Hypothesis is that the longer I run the software, more defects I will find. However, there is no proof for this hypothesis and I’d like to run an experiment. What methodologies can I use? I want to draw a curve with test duration and defect find rate. With this curve, I want you choose the sweet spot that suits my business.
Some basic questions

How can I design the test?
How many samples should I collect?
Techniques to draw the graph?
Any statistical techniques that I can use to back my recommendations?


texts: LSTM Timeseries recursive prediction converge to same value

texts:
I'm working on Timeseries sequence prediction using LSTM.
My goal is to use window of 25 past values in order to generate a prediction for the next 25 values. I'm doing that recursively:
I use 25 known values to predict the next value. Append that value as know value then shift the 25 values and predict the next one again until i have 25 new generated values (or more)

I'm using "Keras" to implement the RNN
Architecture:

regressor = Sequential()
regressor.add(LSTM(units = 50, return_sequences = True, input_shape = (X_train.shape[1], 1)))
regressor.add(Dropout(0.1))

regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.1))

regressor.add(LSTM(units = 50))
regressor.add(Dropout(0.1))

regressor.add(Dense(units = 1))

regressor.compile(optimizer = 'rmsprop', loss = 'mean_squared_error')

regressor.fit(X_train, y_train, epochs = 10, batch_size = 32)

Problem:
Recursive prediction always converge to the some value no matter what sequence comes before.

For sure this is not what I want, I was expecting that the generated sequence will be different depending on what I have before and I'm wondering if someone have an idea about this behavior and how to avoid it. Maybe I'm doing something wrong ...

I tried different epochs number and didn't help much, actually more epochs made it worse. Changing Batch Size, Number of Units , Number of Layers , and window size didn't help too in avoiding this issue.

I'm using MinMaxScaler for the data.


texts: How to extract specific parts of text from a string?

texts:
For example:
Here is a textual input:

"ALL imagery SINCE 1952
20 MULE T
aerial BOOSTER
& Multi-Purpose Neutralizer.MAY BE â€œAHMFLL |-E SWALL """""" -5 NETWT4LBS1 DZ (65 OZ) 1.84 kg

"
Output desired:
4 LBS
65 OZ
1.84 kg
"


texts: I have a question. How is the voting done in random forests. I can't understand rationally, since we have a bootstrap sample drawn, and have built dection trees based on them, where is the new data point taken from to do the voting and extract the results?


texts:
I have a set of images that are considered as good quality image and other set that are considered as bad quality image. I have to train a classification model so that any new image can be said good/bad.
SVM seems to be the best approach to do it. I know how to do it in MATLAB.

But,can anyone suggest how to do it in python? What are the libraries?
For SVM scikit is there, what about feature extraction of image and PCA?


texts: HoG features taking longer time in Google CO-LAB

texts:
I have been doing this for 2 days now.

I want to make a model using sklearn that uses the HoG (Histogram oriented Gradients) features to classify the image (MNIST dataset having 70,000 images both training and testing combine) in Google CO-LAB, but somehow the model is taking much time then it is intended, and I don't know what mistake I am making. Can anyone help me out?

My HoG feature vector size is 6084. I have also divided the training data into train and test sets. I also used both GPU and TPU for the speed enhancement but no positive result has been found.


texts:
I am currently using SVM and scaling my training features to the range of [0,1].
I first fit/transform my training set and then apply the same transformation to my testing set. For example:

    ### Configure transformation and apply to training set
    min_max_scaler = MinMaxScaler(feature_range=(0, 1))
    X_train = min_max_scaler.fit_transform(X_train)

    ### Perform transformation on testing set
    X_test = min_max_scaler.transform(X_test)

Let assume that a given feature in the training set has a range of [0,100], and that same feature in the testing set has a range of [-10,120]. In the training set that feature will be scaled appropriately to [0,1], while in the testing set that feature will be scaled to a range outside of that first specified, something like [-0.1,1.2].

I was wondering what the consequences of the testing set features being out of range of those being used to train the model? Is this a problem?


texts:
I am working on a neural network(Variational Autoencoder) to separate music instruments I don't know how to organize my data I have a folder full of spectrograms of the songs as a mix and the spectrograms of each instrument .. I am so new to coding so I don't know how to organize them. I would really appreciate if someone send me a sample code .. Maybe I can put the instr's path in a dict or something.
I am working with python


texts: Celsius to Fahrenheit conversion simple Neural Network

texts:
This common example is a simple way to dabble into the most basic neural network (i.e. two neurons, one input and one output).

I have been playing around with this using TensorFlow and keras, and I am wondering, it should be very simple for the NN to learn the weight (1.8) and bias (32) with little to no effort.

However, I find that (albeit, using only 13 examples) after running for 500 epochs I can get a weight of 1.8 but the bias is always off (29.2, for example). Obviously there is no activation function, given this linear problem.

My question is, for such a simple case, will a NN be able to solve this problem exactly?


texts:
I have incidents VS normal operation of my working environment. It is a skew dataset. My prediction accuracy is 95%.

Question:
1. Is it common practice among data scientist to accept this prediction?
2. Do I have to rework by resample and balance the skewness and train & test again?

I am sorry to ask for partial opinion based question.


texts: Find boundaries for a smallest mean error

texts:
I am new at deep learning, but willing to learn.

I have this problem.

Have 5 inputs: A,B,C,D,E(columns in CSV) that gives me 1 or 0.
A,B,C,D or E can be between 0 and 100.
For example, a combination of values of these letters will give me 0 or 1.

A  | B | C  | D | E
20, 61, 42, 90, 3 will give me 1
20, 31, 2, 61, 20 will give me 0

Idea is this if A is up to 20 or less, B is up to 61 or less, C is 42 or less, D is
90 or more and E is 3 or more, it will give me 1.

The problem number one is next, I need to find boundaries(Max and Min values) for these numbers. I do not know which column should be Max or Min. So I need to find these boundaries that will give me the smallest mean error.

The problem number two, this is more like next step, sometimes, some of the columns just do not apply to give 0 or 1, they are just relevant, so I need to find relevant columns, and set for example that column to be 100 or less, or 0 or more.

Except writing python code to load CSV, and some research regarding Logistic and Linear regressions, and other algorithms like Nearest Neiber I am kinda stuck, need advice in which direction to go. Any sample math, the code would be amazing.

Hope I have explain it good, really sorry if it is not, please let me know, and I will try to make it clearer.

Thank you.


texts:
In Rshiny i want to wrap the data in the table.How can i do that below is the code

    fluidRow(
        column(4,
               tableOutput('table')
        )

tags$style(HTML('#table{word-wrap: break-word;}')),

But it is not reflecting.I am newbie to UI


texts:
Let we have 2 classes with their densties :
p(x| w1)=$$\tfrac12e^{-|x|} $$
P(x|w2) = $$e^{-2|x-2|}$$
Where x is a one dimension feature vector .
Now it is required to be known where to place the decsision boundry so that the probabilites of errors from both classes are equal to each other. What I have done is that I have equated both probability densty functions and then I have solved this equation to get two solutions one x= 1.10 and x=4.69. Which means that we kan place the boundry at x= 1.10 or x = 4.69. So am I right ? Or did I do something wrong ? Also , can you help me please to sketch both density functions to visualize them ?


texts: Training for Named entity recognition with sparse labels

texts:
I am training an NER pipeline - It was pretrained. My label appears sparsely - Once every 100 or 200 sentences.
Do I have to train my pipelines with ALL sentences or can I speed the training up by only training the relevant sentences?


texts: What is batch size is required for training 30000 images in keras cnn?

texts: I am working on the 30000 images. I kept the batch size as 670 and its showing the 0.76 accuracy. Now I want to improve my accuracy up to 0.92, so for that one what should be the batch size is required?


texts:
I have a regression problem, with a million rows or so, around 10-15 features.
What should work better on that particular setting? Neural network or regular regressors?


texts: Why does the square loss not decrease in deconvolution network?

texts:
I use square loss to measure the difference between the output of the deconvolution network and the input image, use learning rate 0.01 and use SGD to optimize, code is written in tensorflow.

model_params = [["layer1",{"type":"conv_relu", "kernel":[3, 3, 3, 64],  "padding":"same"}],
                         ["layer2",{"type":"conv_relu", "kernel":[3, 3, 64, 64],  "padding":"same"}],
                         ["layer3",{"type":"max_pooling", "ksize":[1, 2, 2, 1],  "padding":"same"}], # ksize":[1, 2, 2, 1],
                         ["layer4",{"type":"conv_relu", "kernel":[3, 3, 64, 128], "padding":"same"}],
                         ["layer5",{"type":"conv_relu", "kernel":[3, 3, 128, 128],  "padding":"same"}],
                         ["layer6",{"type":"max_pooling", "ksize":[1, 2, 2, 1], "padding":"same"}],
                         ["layer7",{"type":"conv_relu", "kernel":[3, 3, 128, 256],  "padding":"same"}],
                         ["layer8",{"type":"conv_relu", "kernel":[3, 3, 256, 256],  "padding":"same"}],
                         ["layer9",{"type":"conv_relu", "kernel":[3, 3, 256, 256], "padding":"same"}],
                         ["layer10",{"type":"conv_relu", "kernel":[3, 3, 256, 256],  "padding":"same"}],
                         ["layer11",{"type":"max_pooling", "ksize":[1, 2, 2, 1], "padding":"same"}],
                         ["layer12",{"type":"conv_relu", "kernel":[3, 3, 256, 1024],  "padding":"same"}],
                         ["layer13",{"type":"conv_relu", "kernel":[1, 1, 1024, 1024], "padding":"same"}],
                         ["layer14",{"type":"deconv_relu", "kernel":[3, 3, 256, 1024],  "padding":"same"}],
                         ["layer15",{"type":"demax_pooling"}],
                         ["layer16",{"type":"deconv_relu", "kernel":[3, 3, 256, 256],  "padding":"same"}],
                         ["layer17",{"type":"deconv_relu", "kernel":[3, 3, 256, 256],  "padding":"same"}],
                         ["layer18",{"type":"deconv_relu", "kernel":[3, 3, 256, 256],  "padding":"same"}],
                         ["layer19",{"type":"deconv_relu", "kernel":[3, 3, 128, 256],  "padding":"same"}],
                         ["layer20",{"type":"demax_pooling"}],
                         ["layer21",{"type":"deconv_relu", "kernel":[3, 3, 128, 128],  "padding":"same"}],
                         ["layer22",{"type":"deconv_relu", "kernel":[3, 3, 64, 128],  "padding":"same"}],
                         ["layer23",{"type":"demax_pooling"}],
                         ["layer24",{"type":"deconv_relu", "kernel":[3, 3, 64, 64],  "padding":"same"}],
                         ["layer25",{"type":"deconv_relu", "kernel":[3, 3, 3, 64],  "padding":"same"}],
                         ]

but the loss did not go down,

2016-03-20 16:36:19.764441: step 0, loss = 54437.99 (24.1 examples/sec; 5.306 sec/batch)
2016-03-20 16:36:42.029972: step 10, loss = 54460.74 (65.3 examples/sec; 1.959 sec/batch)
2016-03-20 16:37:02.488069: step 20, loss = 54398.99 (61.1 examples/sec; 2.096 sec/batch)
2016-03-20 16:37:25.320226: step 30, loss = 54171.65 (61.0 examples/sec; 2.100 sec/batch)
2016-03-20 16:37:45.594397: step 40, loss = 54180.36 (62.6 examples/sec; 2.044 sec/batch)
2016-03-20 16:38:08.296269: step 50, loss = 54444.09 (63.4 examples/sec; 2.020 sec/batch)
2016-03-20 16:38:28.625820: step 60, loss = 54644.72 (61.0 examples/sec; 2.099 sec/batch)
2016-03-20 16:38:50.959914: step 70, loss = 54447.85 (61.6 examples/sec; 2.077 sec/batch)
2016-03-20 16:39:11.174660: step 80, loss = 54334.46 (61.7 examples/sec; 2.074 sec/batch)
2016-03-20 16:41:16.831728: step 140, loss = 54293.51 (67.0 examples/sec; 1.911 sec/batch)
2016-03-20 16:41:38.710691: step 150, loss = 54386.80 (59.9 examples/sec; 2.138 sec/batch)

How to check where go wrong or maybe someone have seen this problem before?


texts: Computing weights in batch gradient descent

texts:
I have a reasonably large set of images that I want to classify using a neural network. I can't fil them all into memory at once, so I decided to process them in batches of 200. I'm using an cross-entropy cost function with a minimization algorithm from numpy.

My question is: is it correct to pass learned weights between batches and use them as a starting point of the minimization? Would this eventually cause my hypothesis to fit all the data, or will each iteration simply re-fit the weights for itself? What is the general approach to such a problem?


texts: Use distribution probability as a feature in ML model

texts:
I built an LSMT model to predict sick cows. I also have risk factors like cow size and height (static risk factor) that I want to combine into the ML model. I found that size is geometrically distributed. My question is how I insert it as a feature to the model? I know that $P(x=K)= p*q^(k-1)$ but I don't know how to combine it as a feature.
Thank you.


texts: How will my CNN results be affected by large discrepancies between the number of samples in some of the classes?

texts: The number of samples in my dataset range between 3800 and 100,000 per class. Was wondering if my neural network will be more biased towards the classes with a higher number of images. I'm trying on the pretrained resnet152V2 model from Keras.


texts: nnet in caret. Bootstrapping or cross-validation?

texts:
I want to train shallow neural network with one hidden layer using nnet in caret. In trainControl, I used method = "cv" to perform 3-fold cross-validation. The snipped the code and results summary are below.
  myControl <- trainControl(## 3-fold CV
    method = "cv",
    number = 3)

  nnGrid <-  expand.grid(size = seq(1, 10, 3),
                         decay = c(0, 0.2, 0.4))

  set.seed(1234)

  nnetFit <- train(choice ~ .,
                   data = db,
                   method = "nnet",
                   maxit = 1000,
                   tuneGrid = nnGrid,
                   trainControl = myControl)

I have few doubts -

The results (attached below) suggests that it performed bootstrapping (25 reps) not cross validation (I was accepting model to be trained only three times for 3-cv for each set of hyperparameter).

I just want to 100% sure on whether model used the original data for training without any pre-processing such as centering and scaling.

I used verboseIter = FALSE in trainControl, but it still prints all the iterations

Are other libraries such as neuralnet, mxnet better than nnet and I can replace them here.

I wan to be sure whether nnet use sigmoid activation function in hidden layer.

Can someone please advise?
No pre-processing
Resampling: Bootstrapped (25 reps)
Summary of sample sizes: 3492, 3492, 3492, 3492, 3492, 3492, ...
Resampling results across tuning parameters:

  size  decay  Accuracy   Kappa
   1    0.0    0.4947424  -0.002382083
   1    0.2    0.5686601   0.141749447
   1    0.4    0.5711497   0.143637446
   4    0.0    0.5076199   0.022765002
   4    0.2    0.7333516   0.468625768
   4    0.4    0.7253675   0.452584882
   7    0.0    0.5002912   0.006079340
   7    0.2    0.7440360   0.488933678
   7    0.4    0.7676500   0.536547080
  10    0.0    0.5064281   0.013648966
  10    0.2    0.7668795   0.535370693
  10    0.4    0.7566465   0.513652332

Accuracy was used to select the optimal model using the largest value.
The final values used for the model were size = 7 and decay = 0.4.


texts: Data quality improvement as a part of preprocessing: Imputation

texts:
I have a python pandas dataframe representing a superset. The data contains a lot of nulls which I want to overwrite with real values.

the superset has:

both numerical and categorical data
some nulls for most attributes
multi class attributes (attributes can have multiple values)
It is not time dependent
each row is a unique person

It would be neat to use machine learning to fill in the nulls, any recommendations on how I can do this?

(I guess that I can tranform the categorical data to numerical if required)


texts: How to penalize a tensorflow dnn regressor model based on specific conditions?

texts: I want to modify the cost function such that the model is penalized more if it gives bad predictions for certain important entities (say top selling products in case of demand prediction). Is it possible?


texts: Is there a network analysis tool built into Orange?

texts: I am interested in a canned or built-in network analysis tool.  Wondering if this is possible with Orange.


texts: What companies would be great for entry level data science/ machine learning programmers to help fight for a good cause?

texts: I am just starting to learn about machine learning, and I have taken some intro classes in MATLAB and Python. I want to know about fulfilling careers and fields programmers can pursue to help me determine my career direction. Thank you for your time!


texts: Right order for Data preparation in Machine Learning

texts:
For the below mentioned steps of data preparation

Outlier detection/treatment
Data imputation
Data scaling/standardisation
Class balancing

There are two sub questions

Should each of these steps performed post test/train split?
Should it be done on test data?

Would appreciate explanation for each step individually.


texts: difference between normal skewed distribution and skewed distribution

texts:
From what I've read normal skewed is a distribution that has all the properties of normal distribution and is skewed:

according to this resource:https://www.statisticshowto.datasciencecentral.com/probability-and-statistics/skewed-distribution/#LogT

one criteria of normal distribution is location parameter(mean) defining peak. Which from my understanding means mode=mean.

So then only difference between normal skewed and skewed distributions are whether or not mode=mean?

Additional Question:
my distribution has mode=1, mean=1.6 thus it is right-skewed (not-normal) distribution. I've read that to fix skewness you can use either logarithms or quantile regression technique, in my case which technique is appropriate?


texts: How to prepare data for Named Entity Recognition with BIO annotation?

texts:
Assume the task here is extracting important facts for resume like a candidate skills and his education etc.., Here is resume is parsed text from pdf or docx resume.

First, I'll obtain skills and education data from various online websites, job portals etc.., the obtaining data will be two text files and in each file every row represent a skills or a university name like below,

skills.txt
___________
c
python
java
node js

education.txt
___________
massachusetts institute of Technology
harvard university

I want to know if these be enough to be able to train a named entity recognition model to recognize skills and education for raw resume text. The data I have is not sentences but just entities. I've read somewhere that we require some context along with the entity for NER model to learn better. Like this example below,

skills.txt
___________
c is used at facebook
python is my favorite programming language

If I use my collected data education.txt to train the modeel and BIO annotate them then it will be like below, it won't have O-Other token.

massachusetts B-EDU
institute I-EDU
of I-EDU
Technology I-EDU

harvard B-EDU
university I-EDU

indian B-EDU
institute I-EDU
of I-EDU
technology I-EDU

But I don't know how to access such data for my resume-extraction problem. How do I proceed further?  How to build an effective NER model for my resume facts identification domain-specific task? Any inputs/suggestions would really help.


texts: How can I get the ImageNet ILSVRC 2012 data used for the classification challenge?

texts:
I would like to see if I can reproduce some of the image net results. However, I could not find the data (the list of URLs) used for training / testing in the ILSVRC 2012 (or later) classification challenges. I only found http://www.image-net.org/download-imageurls.

Where is the data used for the ImageNet ILSVRC 2012 (or later) classification challenge?


texts: Best place for Bengin files for training a classifier?

texts:
Trying to train a Deep neural network classifier, and I got a big data set from virusshare.com, however I also need some normal (non-malware) files. Format must be a PE (x64 or x86).

Does anyone know where to get such a dataset? Or would I have to create my own?


texts:
My wife has seizures a few times a month and takes medication for the seizures once a month.  So I have two sets of dates - the dates of the seizures and the dates she got her medication.

I'm trying to determine whether there is any sort of pattern in the timing of the seizures vs. the medication.  I hope to get some clues as to whether the medication is helping, whether she needs it more often, etc.

I'd appreciate any suggestions as to how to analyze these two sets of dates for any pattern/relationship.  I'm not a data scientist, but I'm pretty good with Excel, so I'm hoping for something I can model in Excel.

Thanks!


texts: Prediction of a dependent variable having comma separated values

texts:
I have a dataset from survey results of Genetic Testing that looks like below:

For more details sample csv file here:

https://drive.google.com/open?id=0BzFPqeNZO-6mM2VwZ1pRQ3NxYTA

I am trying to create a model to predict the column “actionspost” (the extreme right column of the above file). The column currently holds comma separated values like “2, 4, 6”. Each numerical value is a coded action that a patient would take after knowing the results of the genetic test as per the survey filled by the patient. For example, 2 indicates: Change eating habits, 4 indicates: Getting members of his/her family tested and so on. Based on the features in the csv file above, I want to create a model that would help me predict the “actionspost” values.

One way I thought, I could do this was break down comma separated values in “actionspost” to create duplicate rows for same customer Ids holding only one “actionspost” value at a time (For example: There would be 4 rows for customer ID: C00003 with first row having “actionspost” value of 2, second row with “actionspost” value 8, third with 9 and fourth with 10. Then create subset of the data set for each unique value of “actionspost”. This time, the dependent variable will not be actionspost but something like “isactionpost2?” (Yes =1 and No = 0). And train models for each of this subset for different values within the “actionspost” column.

I personally don’t like the above method as it too cumbersome and in my opinion, not the optimal one too. I was wondering if there could be a better way to address such scenario? My end goal is to try to train different models like Decision Trees, Naïve Bayes, and Neural Nets and check which one leads to better predictability.


texts: How to Precondition A Fully Convolutional Neural Network

texts:
Lets say I am creating a fully convolutional classifier for identifying different types of animals. Now lets assume that I know what part of the animal kingdom the animal belongs to beforehand (mammal, reptile, amphibian .. etc). How would I present this extra information to the fully convolutional network?

Any ideas or research would be very helpful.

Edit*
More specifically I am concerned with domain adaptation as shown here, In this example I am using a fully convolutional network without any pooling or striding, where the output is the same size as the input (i.e. an image transformed from one domain to another). In this respect I want to be able run the domain adaptation on images of any size. Adding extra channels to the input seems to make the most sense, however if I am one-hot encoding a bunch of categorical variables, this will consume a lot of memory.


texts: What is difference between attention mechanism and cognitive function?

texts: How are attention mechanism used in different deep learning algorithms different from the cognitive function attention?


texts: Training an AI to play Starcraft 2 with superhuman level of performance?

texts:
I'm interested in working on challenging AI problems, and after reading this article (https://deepmind.com/blog/deepmind-and-blizzard-open-starcraft-ii-ai-research-environment/) by DeepMind and Blizzard, I think that developing a robust AI capable of learning to play Starcraft 2 with superhuman level of performance (without prior knowledge or human hard-coded heuristics) would imply a huge breakthrough in AI research.

Sure I know this is an extremely challenging problem, and by no means I pretend to be the one solving it, but I think it's a challenge worth taking on nonetheless because the complexity of the decision making required is much closer to the real world and so this forces you to come up with much more robust, generalizable AI algorithms that could potentially be applied to other domains.

For instance, an AI that plays Starcraft 2 would have to be able to watch the screen, identify objects, positions, identify units moving and their trajectories, update its current knowledge of the world, make predictions, make decisions, have short term and long term goals, listen to sounds (because the game includes sounds), understand natural language (to read and understand text descriptions appearing in the screen as well), it should probably be endowed also with some sort of attention mechanism to be able to pay attention to certain regions of interest of the screen, etc. So it becomes obvious that at least one would need to know about Computer Vision, Object Recognition, Knowledge Bases, Short Term / Long Term Planning, Audio Recognition, Natural Language Processing, Visual Attention Models, etc. And obviously it would not be enough to just study each area independently, it would also be necessary to come up with ways to integrate everything into a single system.

So, does anybody know good resources with content relevant to this problem? I would appreciate any suggestions of papers, books, blogs, whatever useful resource out there (ideally state-of-the-art) which would be helpful for somebody interested in this problem.

Thanks in advance.


texts: Under what circumstance is lemmatization not an advisble step when working with text data?

texts:
Disregarding possible computational restraints, are there general applications where lemmatization would be a counterproductive step when analyzing text data?

For example, would lemmatization be something that is not done when building a context-aware model?

For reference, lemmatization per dictinory.com is the act of grouping together the inflected forms of (a word) for analysis as a single item.

For example, the word 'cook' is the lemma of the word 'cooking'. The act of lemmatization is, for example, replacing the word cooking with cook after you have tokenized your text data. Additionally, the word 'worse' has 'bad' as its lemma, and as the previous example replacing the word 'worse' with 'bad' is the action of lemmatization.


texts:
I've got survey data that resembles:

|-------------| Q1a | Q1b | Q1c | Q2a | Q2b | Q2c | Classification
| Respondent  | 1   | 0   | 0   | 1   | 0   | 0   | Red
| Respondent  | 0   | 0   | 1   | 1   | 0   | 0   | Green
| Respondent  | 0   | 1   | 0   | 0   | 0   | 1   | Yellow

I am trying to predict the classification for new respondents. Currently I'm using a Naive Bayes, and getting pretty bad accuracy (~20%). I don't have much training data, and the training data is hand scraped from non-standard sources (internal company procedures are a mess here).

I'm looking for other ways to predict the classification.

I'm thinking about assigning weights to each question, and magically predicting the result based on those, somehow. Although I don't really know where to start learning about how to do that, and whether it's appropriate for this data. I have very little background in this :(

Any ideas or tips on predicting the classification column with no training data?


texts: ML/Statistical Model to Analyse the Distribution

texts:
Consider a Sample Data-set provided below;

|ShopID|    |Transactions|    |dist_to_shop|
   S1           15478               0
   S2           12345              0.41
   S3           17865              0.11
   S4           35479              0.57
   S5           74589              0.35

The data-set consist of ShopID, Transactions and dist_to_shop (In Meters) fields. Assuming all the Shops belong to one retailer, I would like to find out the distribution of Transaction/People Visits to the other shops, by assigning weights/business rules on the basis of the distance.

For Example, the weights can be given as;

0-200 Meters = 40%
201-400 Meteres = 30%
401 & Above = 20%

Question - What would happen if the ShopID S3 is closed and the possible transactions/people_count splitting or distributing in other shops based on the distance weights provided. It's similar to What-If Analysis. I would like to know the spread of the Transactions if a Shop Closes down.

My data-set almost looks similar to the one I have provided above. I would like to know the best approach to solve this scenario. Which ML/Statistical Model will be best suitable.

Any inputs will prove really valuable.

Edit: Changes made on 25/Feb/2019


texts: Are there any meanings for the dimensions of a t-sne embedding? Like with PCA we have this sense of linearly transformed variance maximizations but for t-sne is there intuition besides just the space we define for mapping and minimization of the KL-distance?


texts: DIfferent learning rates converging to same minima

texts:
I am optimizing some loss function using Gradient Descent method. I am trying it with different learning rates, but the objective function's value is converging to same exact point.

Does this means that I am getting stuck in a local minima?, because the loss function is non-convex so it is less likely that I would converge to a global minima.


texts: I have to build one class naive Bayes method for outlier detection based on the likelihood probabilities. I tried to create it using sklearn GaussianNB but it is for multiclass classification. Is there any way to do it in python or Matlab?


texts: grid search - optimal weighting of classifiers

texts:
I am using three different of the shelf classifiers. It's a three class classification task.
I want to calculate the optimal weights (c1weight, c2weight, c3weight) for each classifier (real task more classifiers and also weights for each class).
Maybe simple grid search approach or sklearn ensemble classifier could do that.
vc = VotingClassifier(estimators=[('gbc',GradientBoostingClassifier()),
                   ('rf',RandomForestClassifier()),('svc',SVC(probability=True))],
                   voting='soft',n_jobs=-1)

params = {'weights':[[1,2,3],[2,1,3],[3,2,1]]}
grid_Search = GridSearchCV(param_grid = params, estimator=vc)
grid_Search.fit(X_new,y)
print(grid_Search.best_Score_)

I don't understand how to implement this for the following code.
def get_classification(text, c1weight, c2weight, c3weight):

  prediction1 = classifier1.predict(text)
  if prediction1 = 1:
    class1 =+ 1 * c1weight
  elif prediction1 = 2:
    class2 =+ 1  * c1weight
  else:
    class3 =+ 1  * c1weight

  prediction2 = classifier2.predict(text)
  if prediction2 = 1:
    class1 =+ 1 * c2weight
  elif prediction2 = 2:
    class2 =+ 1  * c2weight
  else:
    class3 =+ 1  * c2weight

  prediction3 = classifier3.predict(text)
  if prediction3 = 1:
    class1 =+ 1 * c3weight
  elif prediction3 = 2:
    class2 =+ 1  * c3weight
  else:
    class3 =+ 1  * c3weight

  if class1 > class2 and class1 > class3:
    return ("class1",class1)
  elif class2 > class1 and class2 > class3:
    return ("class2",class2)
  else:
    return("class3",class3)

c1weight = 0.5
c2weight = 0.7
c3weight = 0.4

for i, row in df_raw.iterrows():
    classification = get_classification(df_raw.at[i, 'text'],c1weight, c2weight, c3weight)
    df_raw[i,'classification'] = classification

score = get_accuracy(df_raw['classification'],df_raw['label'])

SOLVED: This sample code helped me to understand it
def your_function(number):
    print(number)

from sklearn.model_selection import ParameterGrid
param_grid = {'param1': [1, 2, 3]}

grid = ParameterGrid(param_grid)

for params in grid:
    your_function(params['param1'])


texts: I keep seeing LeNet used to referring to a convolution network? I am wondering why LeNet is called LeNet? Is it the abbreviation of anything? Is there a difference between LeNet and convolutional neural network? Thanks!


texts: How many channels do ultrasound images have?

texts:
I have some ultrasound images. When I read them using cv.imread and check their shape or check them with ndim, it shows 3 channels:
im = cv.imread(data_path+'a1.jpg')
print(im.shape)
im = np.array(im)
print(im.ndim)

the output is:
(154, 139, 3)
3

But when I open them in photoshop and check their mode in Image > Mode, it shows grayscale with 8 bit. Also in windows properties > detail, tab shows 8 bit:

Dose anyone have any idea about that?