texts: build biased image dataset for emotion analysis

texts:
This is a pre-project question.
I would like to find or build a biased dataset to demonstrate what happens if training data are biased (biased distributed ethnicity for exemple).
I try this for the dataset (also this one) and Azure custom vision for the training.
My question is: if I remove one part of the images, would it be enough to bias the dataset and make the prediction correct only for subsets of the original dataset?


texts: Machine learning system which can learn from data and human rules

texts:
I have a data set and few human rules. I want to learn the machine learning system. One of the paper I came across with is:

Harnessing Neural Network

Above paper trained some teacher network to learn the system in order to improve the accuracy but they still require enough data.
But my requirement is different. It may be possible that we have very few data points corresponding to a rule. How can I design a system which can utilize the both learning from data set and human rules.


texts: Intensity image to RGB for transfer learning

texts:
My goal is to use a pre-trained model with an intensity-based image.
Most pre-trained models expect RGB (int) format as the input image.
An easy workaround is to duplicate the intensity channel 3 times in order
to obtain a greyscale image.
However, my intensity image has a large scale (not 0 - 255), so this solution will not work. I could squeeze the range, at the cost of information.
Fortunately, RGB space is 256256256. So it is the space to contain a very large intensity range. But what is the "best" way to do so?
Actually, I use matplotlib to apply color map on my image, which kinda does what I want but I'm not sure if that's correct, nor which colormap is the best.


texts: How to speed up passing of images to a GPU

texts:
I am using Ubuntu 16.04 LTS installed on a 4TB HDD. I am working with a large dataset (more than 30 GB and 150,000 images). I have a single 11GB GTX 1080 Ti GPU card on my system. I am training a mask RCNN written in Keras. I noticed that passing the entire dataset of 150,000 images to a single GPU from the HDD will take few weeks at least to process a single epoch. Hence I want to speed up passing an image to a GPU.

So I bought a 256 GB SSD, mounted it and copied only the images directory to the SSD. Then I began to train the Keras model. All python files including the Anaconda install are on the HDD. I noticed that merely moving the images to SSD did not speed up the training. Does anyone know how to speed up the training process by sending images from the SSD to the GPU? Do I need to move the python files and anaconda install to the SSD as well? I do not wish to reinstall the entire OS on the SSD.


texts: Convolution Neural Networks on microcontrollers

texts:
I am wondering if an already trained convolutional neural network can be represented as a formula just like a perceptron can ($x_1w_1 + x_2w_2 + \ldots =$ PREDICTION). I know that the formula could be a lot more complicated like this, but in general would it be possible to train a CNN in Python or Matlab, get the weights and create an arithmetic, exponential, whatever formula made with those acquired weights?

I want to do this because I am trying classify in a PIC32 which I think does not have the capacity to perform a CNN within itself.

Or do you know if there is a "c library" that can do CNNs?


texts: What's the difference between Attention vs Self-Attention? What problems does each other solve that the other can't?

texts: As stated in the question above..is there a difference between attention and self attention mechanism ? Also additionally can anybody share with me tips and tricks about how self attention mechanism can be implemented in CNN?


texts:
Trying to improve my chat App:

Using previous (pre-processed) chat interactions from my domain, I have built a tool that offers the user 5 possible utterances to a given chat context, for example:

  Raw: "Hi John."

Context: hi [[USER_NAME]]
Utterances : [Hi ,Hello , How are you, Hi there, Hello again]

Of Course the results are not always relevant, for example:

  Raw: "Hi John. How are you? I am fine, are you in the office?"

Context: hi [[USER_NAME]] how are you i am fine are you in the office
Utterances : [Yes, No , Hi , Yes i am, How are you]

I am using Elasticsearch with TF/IDF similarity model and an index structured like so:

{
  "_index": "engagements",
  "_type": "context",
  "_id": "48",
  "_score": 1,
  "_source": {
    "context": "hi [[USER_NAME]] how are you i am fine are you in the office",
    "utterance": "Yes I am"
  }
}

Problem: I know for sure that for the context "hi [[USER_NAME]] how are you i am fine are you in the office" the utterance "Yes I am" is relevant, however "Yes" , "No" are relevant too because they appeared on a similar context.

Trying to use this excellent video, as a starting point

Q: How can I measure precision and recall, if all I know (from my raw data) is just one true utterance?


texts:
Suppose I am processing 3rd party vendor error log files that I am unable to change the schema from the source. And I am trying to predict a label.

Once logs are collected and I have all I need, in varying log formats, with each missing random information compared to the others.

-------------------------------------------------------
| vendor | variable_one | variable_two |variable_three|
-------------------------------------------------------
|    1   |       -      |"some string" |     NULL     |
-------------------------------------------------------
|    3   |      GET     |"some string" |     1270     |
-------------------------------------------------------
|  ...   |     POST     |    NULL      |      760     |
-------------------------------------------------------

Even after all the preprocessing and data transformations to change labels to Binary, and Categorical values and such. After cleaning, feature selection and training. Having a model trained, evaluated, and tested. (label not depicted)

-------------------------------------------------------
| vendor | variable_one | variable_two |variable_three|
-------------------------------------------------------
|    1   |      GET     |"some string" |        0     |
-------------------------------------------------------
|    3   |      GET     |"some string" |     1270     |
-------------------------------------------------------
|  ...   |     POST     |     ""       |      760     |
-------------------------------------------------------

The questions are:

When I go back to send my vendors logs to through the model, I am still sending it through with, potentially, missing values from what was altered, or selected as a feature for the model.

Is the model then useless?
Do the transformations made in data cleansing need to be applied to each log entry that comes through, somewhere upstream?
Assuming the model only knows "nice pretty" cleaned data, and I send log entries with missing values. Is the model making assumptions or less accurate prediction?


texts: Machine learning model using keywords for binary decision

texts:
I have a some experience from Uni with convolutional NN and edge detection, but haven't much explored the other types of machine learning models.

I was wondering if there might be one that is suited for being able to use certain, textual keywords (for example, words that were parsed from internet news sources) in order to arrive at some binary, yes/no decision?


texts:
What is the main diffrence between flow_from_directory VS image_dataset_from_directory in keras?
which one should I use?


texts:
I'm struggling with my project, based on kfda package.
The Code is here.
data1=pd.read_csv('one.csv', header=None)
data1
data1=data1[0].str.split(expand=True)
data1

data2=pd.read_csv('two.csv', header=None)

dataset=pd.concat([data1, data2], axis=1)
dataset

data=dataset.values
X,y = data[:,:-1], data[:,-1]
print(X.shape, y.shape)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=1, stratify=y)
print(X_train.shape)
print(y_train.shape)

from kfda import Kfda

cls = Kfda(kernel='rbf', n_components=1, gamma=0.2)
print('Fitting...')
fitting=cls.fit(X_train,y_train)

test_score = cls.score(X_test, y_test)
print(f'Test Score: {test_score}')
train_score = cls.score(X_train, y_train)
print(f'Train Score: {train_score}')

The error I get here:
fitting=cls.fit(X_train,y_train)

As a messege I get
Traceback (most recent call last):
  File "D:\Python\task.py", line 67, in <module>
    fitting=cls.fit(X_train,y_train)

Could you please be so kind to have a short look, what I can do with that.
Thanks a lot in advance!


texts: Is it ideally correct to benchmark neo4j as graph processing platform?

texts:
I would like to know if neo4j can be considered a graph processing platform, even though i know that:

neo4j: is a graph database management system developed by Neo Technology, Inc. Described by its developers as an ACID-compliant transactional database with native graph storage and processing.

graph processing platform: is a platform used for processing graphs, by applying global algorithms towards large graphs (used more in OLAP scenario).

Based on this paper http://www.ds.ewi.tudelft.nl/~iosup/perf-eval-graph-proc14ipdps.pdf, neo4j is benchmarked as a processing platform.

So my question is, if its correct to put neo4j in the group of processing platform like pregel and giraph.


texts: Normalization of possibly not fully representative data

texts:
I am trying to train a classification RNN model on a sequence of table medical data, but I stuck with the normalization problem. I realized that I cannot simply use MinMaxScaler, because of 3 problems:

outliers, but I could fight them or use RobustScaler instead.
I am not sure that some features in my dataset include all possible ranges. Like I have max(feature_A) == 10, but with the data update, it could become 20. And if I'll preprocess data the same way I will get bad prediction results.
Some features do not have a limit at all and will only increase with time, like how many years patients were treated, for example. I could suppose that this value is !>100years, for example, but if my mean value is 10 years, it will squeeze feature values a lot.

My dataset is pretty large, like millions of observations, thus there is a pretty good chance that it is representative, though. But I am concerned with the small-time range, like all those observations are for the 2 years only, thus, some feature values (like how many years patients were treated) could still grow their bounds.
How should I handle this?
My concerns example:
import pandas as pd
from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler()

#### like, initial state
df1 = pd.DataFrame({'A': [1, 2, 3, 4, 5], 'B': [10, 40, 60, 80, 100]})
""" output:
   A    B
0  1   10
1  2   40
2  3   60
3  4   80
4  5  100
"""

scaler.fit_transform(df1)
""" output:
array([[0.        , 0.        ],
       [0.25      , 0.33333333],
       [0.5       , 0.55555556],
       [0.75      , 0.77777778],
       [1.        , 1.        ]])
"""

#### new data arrived, while preprocessing is the same
df2 = pd.DataFrame({'A': [1, 2, 3, 4, 5, 10, 10], 'B': [10, 40, 60, 80, 100, 120, 140]})
""" output:
    A    B
0   1   10
1   2   40
2   3   60
3   4   80
4   5  100
5  10  120
6  10  140
"""

# now 5 in "A" scaled to 0.4 instead of 1, same in "B"
scaler.fit_transform(df2)
""" output:
array([[0.        , 0.        ],
       [0.11111111, 0.23076923],
       [0.22222222, 0.38461538],
       [0.33333333, 0.53846154],
       [0.44444444, 0.69230769],
       [1.        , 0.84615385],
       [1.        , 1.        ]])
"""

PS: I've duplicated this question in different communities (question in ai got most of views):

https://stats.stackexchange.com/questions/523097/normalization-of-possibly-not-fully-representative-data
https://ai.stackexchange.com/questions/27678/normalization-of-possibly-not-fully-representative-data


texts: K-Means Clustering Profile Plot & Data Normalization

texts:
I am new to k-means clustering and I am working on a project on cryptoanalysis. I have a few questions and I hope to get some help here.

I have four variables and my variables data values can range from -100% to 100% (as of now, the data I used currently range from -40% to 5%), do I still need to normalize the data? If yes, do I use z-score normalization?
If I did to normalize data in market capitalization of an asset, how do I go about doing it?
Usually, in examples, the k-means clustering runs on 2 variables and you can plot the clusters in the x-axis and y-axis. In this case, I have 4 variables how do I go about plotting the clusters in a chart?
I have tried to run k-means with 4 clusters and plot 2 profile plot: 1st plot). My data are not normalized before running k-means, 2nd plot) My data are all z-score normalized before running k-mean. The 2nd plot looks wrong as cluster 2 & 4 looks like a straight line and the two profile plot looks very different after I did the z-score normalization. Did anyone get any advice?

1st plot (No normalization)

2nd plot (Z-Score normalization)


texts: What is the purpose of multiple neurons in a hidden layer?

texts:
On the surface, this sounds like a pretty stupid question. However, i've spent the day poking around various sources and can't find an answer.

Let me make the question more clear.

Take this classic image:

Clearly, the input layer is a vector with 3 components. Each of the three components is propagated to the hidden layer. Each neuron, in the hidden layer, sees the same vector with 3 components -- all neurons see the same data.

So we are at the hidden layer now. From what I read, this layer is normally just ReLus or sigmoids.

Correct me if I'm wrong, but a ReLu is a ReLu. Why would you need 4 of the exact same function, all seeing the exact same data?

What makes the red neurons in the hidden layer different from each other? Are they supposed to be different? I haven't read anything about tuning or setting parameters or perturbing different neurons to have them be different. But if they aren't different...then what's the point?

Text under the image above says, "A neural network is really just a composition of perceptrons, connected in different ways." They all look connected in the exact same way to me.


texts:
GPS Data Preprocessing

I got  gps traces of busses which traveled from one place (bus station) to another. This file contains relevant(eg:- Data recorded between the starting bus station and ends with the destination stop) and irrelevant GPS traces(eg- traces recorded from bus station to the home of the bus driver). All the data of the month is in the same file. Looking for a efficient preprocessing technique to separate journeys from the file. The data fields in the files are "timestamp", "lat","lon".  Can somebody suggest any techniques for this preprocessing step?


texts: How can I access dataset from Nasa website

texts: So Now I was working on my project ("Failure prediction models in FoF") and the Tool I will be using is RapidMinder but I need to have a dataset to run the tool on, I found a nice description of dataset that could be helpful on Nasa website https://data.nasa.gov/external-dataset?datasetId=dc5d-zf5v , However I do not know how to access the data itself in form of CSV file.


texts: Value of features is zero in Decision tree Classifier

texts:
I used CountVectorizer and TfidfVectorizer seperately to vectorize text which is 100K reviews and passed the vector data to a Decision tree Classifier. Upon using _feature_importances__ attribute of Decision tree Classifier, the feature importance values for all my feature are just 0.0. But with the same dataset, I'm able to find feature importances for naives bayes and logisitic regression by using feature_prob for naive bayes and coef_ attributes for logisitic regression.

Other things I tried:
1. I tried changing ngram_range in countvectorizer
2. I tried limiting/not limiting min_df and max_feature parameters passed in countvectorizers
But couldn't make it work.

Any help is appreciated

Code:

positivereviews = df[df.Score == 1]
negativereviews = df[df.Score == 0]

countvect = CountVectorizer(stop_words='english')

positivebow = countvect.fit(positivereviews.CleanedText[0:100000])

pos_xtrain = positivebow.transform(positivereviews.CleanedText[100000:200000])

pos_y = positivereviews.Score[100000:200000]

clf = DecisionTreeClassifier(max_depth=3, min_samples_split=2)

clf.fit(pos_xtrain, pos_y)

def show_most_informative_features(vectorizer, clf, n=20):
    feature_names = vectorizer.get_feature_names()
    coefs_with_fns = sorted(zip(clf.feature_importances_, feature_names))
    top = coefs_with_fns[:-(n + 1):-1]
    for (coef_1, fn_1)in top:
        print ("\t%.4f\t%-15s" % (coef_1, fn_1))

Counter(clf.feature_importances_)
Counter({0.0: 63514})

1) positivereviews is the dataframe which has features CleanedText and Score
where CleanedText is text which is preprocessed and Score is either 1 in this set since I splitted positive and negative review using Score

2) I also searched online for this problem, but couldn't find any instance of this issue

EDIT1: can it be because since we are dealing with categorical feature in this problem, I might be overfitting badly and hence I do not see any value for any features?

Thanks in advance!!


texts: How to get data back into two separate audio files after successfully applying kmeans clustering on an audio file?

texts:
Okay so I wrote a very simple python code to read a wav file, get the mfcc features and then use kmeans clustering on the features. The hello.wav file has two different people saying hello at the same time, I wanted to output two separate voices

from python_speech_features import mfcc
from scipy.io import wavfile
from sklearn.cluster import KMeans
sampfreq, data = wavfile.read('hello.wav')

mfcc_feat = mfcc(data,sampfreq)

kmeans = KMeans(n_clusters=2)
kmeans.fit(mfcc_feat)
print(kmeans.cluster_centers_)
print(kmeans.labels_)

Now I got the labels and cluster centers but how do I get back my two separate sound samples using this?


texts: Different results for LogisticRegression on python 2.7 and 3

texts:
I have different results for the same kernel on python 2.7 (local machine) and python3 (the system running on kaggle) for LogisticRegression. How it is possible?

Here my results from my local machine:

Here from the kaggle notebook:

The amount of data is different, because I split a bit more for the train data, but the predictions are totally different.  Can this be because of the versions of python?


texts:
Seaborn barplot has three parameters.

  x, y, hue : names of variables in data or vector data, optional

Question

What is hue? It seems the attribute to plot but why it is called "hue" because when I googled, the result is about color?

Google

Hue - Wikipedia

  Hue is one of the main properties (called color appearance parameters) of a color, defined technically (in the CIECAM02 model)


texts: tf.keras: val accuracy on mnist dataset drops to random guess

texts:
For transfer learning purposes I changed mnist into 224x224 px dataset and applied skeletonizing algorithms.

I want to train MNIST dataset on "squeezenet" like CNN, which can be found on github.

The issue is that when training from scratch (random weights) after some time accuracy drops to random guess, see:

Minimal code for reproducibility:

train_datagen = ImageDataGenerator(
rotation_range=60, rescale=1.0/255
)

test_datagen = ImageDataGenerator(rescale=1.0/255)

train_generator = train_datagen.flow_from_directory(
os.getcwd() + str(config.NEW_DATA_MNIST) + str(Path("/train")),

    target_size=(224, 224),

    class_mode="categorical",

    batch_size=32
)

validation_generator = test_datagen.flow_from_directory(os.getcwd() + str(config.NEW_DATA_MNIST) + str(Path("/test")),

    target_size=(224, 224,

    class_mode="categorical",

    batch_size=32
)

base_model = SqueezeNet(
input_shape=(224, 224, 3),
include_top=False,weights=None)

x = base_model.output

x = GlobalAveragePooling2D()(x)
predictions = Dense(10, activation='softmax', name='predictions')(x)

model = Model(inputs=base_model.input, outputs=predictions)

model.compile(
optimizer='sgd',
 loss='categorical_crossentropy',
 metrics=['accuracy']
)

history = model.fit(self.train_generator,

     epochs=self.simulation_config.epochs,
     validation_data=self.test_generator)

This is a minimal example, so hopefully it is understandable.

To further clarify the problem, I tried multiple optimizers ('sgd', 'adam', 'adadelta'). I use tf.keras and tensorflow-gpu==1.14.0. Anybody has some ideas how to fix this issue?


texts: How to set class weights for imbalanced classes in Keras?

texts:
I know that there is a possibility in Keras with the class_weights parameter dictionary at fitting, but I couldn't find any example. Would somebody so kind to provide one?

By the way, in this case the appropriate praxis is simply to weight up the minority class proportionally to its underrepresentation?


texts: Difference between Gradient Descent and Normal Equation in Linear Regression

texts:
Hi I am new to Linear Regression.
I want to know

  what is the difference b/w Gradient Descent and Mean Square Error in
  Linear Regression using machine learning?

And

  When to use Gradient Descent and Mean Square Error in Linear
  Regression using machine learning?
  Or

  When to use which algorithm in Linear Regression.?

Can anyone explain.?


texts: What should be the ratio of True vs False cases in a binary classifier dataset?

texts:
I am using a CNN for sentiment analysis of news articles. It is a binary classification with outputs: Interesting & Uninteresting.
In my dataset, there are around 50,000 Uninteresting articles and only about 200 Interesting articles. I know the ratio is badly skewed.

My question is what should be the ratio in such a scenario.
One approach that I want to try is to cluster the Uninteresting news
articles and take a sample from each cluster for training. Is there
a better approach?


texts: Matrix factorization how to initialize weights and biases?

texts:
I have a matrix factorization and I'm wondering how I should initialize its weights and biases.
When getting prediction (recommendation), after computing a dot product and adding bias I want to use sigmoid function on that to get value from 0 to 1.
But when introducing a sigmoid here I also introduce a possibile vanishing/exploding gradient problem. For that I think that weights can be initialized with xavier function. But what aboud biases? Should I just use uniform distribution from (-0.01, 0.01) for example?


texts: Is it possible for a neural net to score as high as a different form of supervised learning?

texts:
I've been working with the Adult Census Income dataset from UCI
http://archive.ics.uci.edu/ml/datasets/adult

I've created two different models, one using a gradient boosted classifier with sklearn, and one with a neural net using Keras/Tensorflow.

So I'm not interested in code hints or anything, but I have a general question about machine learning-I have a significantly higher accuracy using the gradient boosted classifier than I do with the neural net.

In general, is it possible for my (or any, really) neural net to reach the same accuracy as any other kind of supervised learning? Does it just take a lot of hard work and elbow grease to tune the neural net well enough? I'm working with the same dataset, using the same feature engineering for both the NN and the gradient boosted classifier.


texts: Can Adagrad be used to optimize non-differentiable functions?

texts:
I am reading a book (TensorFlow For Dummies, Matthew Scarpino), and here it says:

  Adagrad methods compute subgradients instead of gradients. A subgradient is a generalization of a gradient that applies to nondifferentiable functions. This means AdaGrad methods can optimize both differentiable and nondifferentiable functions.

But I haven’t seen such a claim anywhere else. So I want to know:

Can we really optimize non-differentiable loss functions just because we are using Adagrad?


texts: How does stacking help Bias and Variance?

texts:
How does stacking help in terms of bias and variance?
I have a hunch that stacking can help reduce bias but i am not sure, could someone refer to a paper?


texts: Sklearn - Override random_state=None by default

texts:
Many scikit-learn and pandas objects/functions use random_state=None as a default parameter. How can it be overridden to random_state=100 by default for all objects without manually editing the random_state for each object?

make_blobs(n_samples=100, n_features=2, centers=3, cluster_std=1.0,
           center_box=(-10.0, 10.0), shuffle=True, random_state=None)


texts: Adapting Neural Network to new domain without labels

texts:
Is there an approach for the following problem:

Lets say, I trained a neural network on a big dataset for categorizing different fruits in $k$ classes. Afterwards I got a nice model, which performs very well.

Now I want to use the model for categorizing fruits in the corresponding $k$ classes, as it was planned beforehand. Unfortunately the fruits I want to categorize now are all not ripe yet, but my training set consisted only of ripe fruits. Furthermore I have some pictures of these not ripe fruits, but no labels.

How can I adapt my neural network to these slightly different domain with my pictures of not ripe fruits (and no labels!). Performance on the old task does not matter. The only thing I want, is categorizing not ripe fruits.

My only Idea now is to use virtual adversarial training (VAT) for the unlabeled pictures.


texts:
I have a system which sends invitations to users to participate in online questionnaires and want to use machine learning in order to predict the likelihood of fulfilling the questionnaires in a predefined time ( i.e. within 1 day , 2 day , 3 days ,a week, 2 weeks, etc) based on various feature related to the users to whom the invitations are sent to , details of the questionnaires ( i.e. how long are they , their topic ,etc.) , other contextual data ( time of day , day of year , in which media the invitation are sent - i.e. sms / email etc).
I can train with positive examples ( invitations that were responded to by users ) and negative examples ( invitations that WERE not responded to ) , however , I'm not sure how to take into consideration the "predefined time" into the feature vector. for example, should I simply include a feature of "days since invitation was sent" and in the positive examples include the time , and in negative examples replicate each example X each of the predefined times to indicate that the users didn't respond at all ?

Any advise would be welcome  !


texts: Neural network with flexible number of inputs?

texts:
Is it possible to create a neural network which provides a consistent output given that the input can be in different length vectors?

I am currently in a situation where I have sampled a lot of audio files, which are of different length, and have to train a neural network provides me the desired output given a certain input.  I am trying to create a regression network that can generate MFCC feature, given samples of an audio file, which are of different length, which makes different numbered input.


texts: Experimental design - hyperparameter optimization

texts:
I'm reading a paper by Bergstra and Bengio (2012) on random search for hyperparameter optimization. I'm confused by their graph and explanation in Section 2.2:

"Figure 2 illustrates the results of a random experiment: an experiment of 256 trials training neural networks to classify the rectangles data set. Since the trials of a random experiment are i.i.d., a random search experiment involving $S$ i.i.d. trials can also be interpreted as N independent experiments of $s$ trials, as long as $sN \le S$."

That part I think I understand. Here's my take on it:

If I draw from a uniform(0, 1) random variable 100 times, then I have 100 i,.i.d. samples. That's the same as if I had drawn 10 times each from 10 separate uniform(0, 1) random variables.

In their figure (below), they say they had 256 trials. I think they chose 256 possible values of a hyperparameter and trained a neural network for each value.

In the caption for the figure, they say they had "256 trials of a random experiment...optimizing a neural network....At horizontal axis position 8, we consider 256 trials to be 32 experiments of 8 trials each".

This implies that at horizontal axis position 1, they would have 1 experiment of 256 trials.

My question is: did they train those 256 models and then break up the results into smaller chunks? Like this:

Experiment 1: $[\lambda^1, \ldots, \lambda^{256}]$

Experiment 2: $[\lambda^1, \ldots, \lambda^{128}], [\lambda^{129}, \ldots, \lambda^{256}]$

...

with value $\lambda^{129}$ from Experiment 1 equal to value $\lambda^{129}$ in Experiment 2?

Or did they do something else?


texts: Convoluational layer is larger than max pooling layer

texts:
I have been trying to convert an implementation of Yolo written in dark-net found here.

I'm having issues converting the original source code as the layers implemented in an alternative library seem to have an implementation gap as tiny_dnn requires more parameters to be set...

From the link above, line 44:

let output = input
    --> Resize(width: YOLO.inputWidth, height: YOLO.inputHeight)
    --> Convolution(kernel: (3, 3), channels: 16, activation: leaky, name: "conv1")
    --> MaxPooling(kernel: (2, 2), stride: (2, 2))
    --> Convolution(kernel: (3, 3), channels: 32, activation: leaky, name: "conv2")
--> MaxPooling(kernel: (2, 2), stride: (2, 2))

I'm trying to translate the above into c++ but I am missing parameters, as the swift code above must infer a number of variables... surely... as it seems to not require stating the following variables:

Window size,
bias,
wstride,
hstride,
padtype,
pooling size

Here is the code I'v tried in c++:

int w=416;
int h=416;
net
<< convolutional_layer(w, h,3,16,16)
<< leaky_relu_layer(0.1F)
<< max_pooling_layer(shape3d(w , h , 3),2,2, core::default_engine());

however that results in an error between the leaky_relu_layer and the max_pooling as it says the output of the leaky relu layer does not match the input of the max pooling layer.

The parameters of the c++ are:

  convolutional_layer(size_t in_width,
                      size_t in_height,
                      size_t window_size,
                      size_t in_channels,
                      size_t out_channels,
                      padding pad_type             = padding::valid,
                      bool has_bias                = true,
                      size_t w_stride              = 1,
                      size_t h_stride              = 1,
                      core::backend_t backend_type = core::default_engine())

max_pooling_layer(const shape3d &in_shape,
                    size_t pooling_size,
                    size_t stride,
                    core::backend_t backend_type = core::default_engine())

Could anyone possibly explain why the output of my convolutional/leaky relu results in a different sized output than the input of the max pooling layer?


texts: Difference between Q-learning and G-learning in Reinforcement Learning?

texts:
What is the difference between Q-learning and G-learning in Reinforcement Learning? Please explain with formulas.
An example source:

Instead of relying on a utility of consumption, we present G-Learner: a reinforcement learning algorithm that operates with explicitly defined one-step rewards, does not assume a data generation process, and is suitable for noisy data. Our approach is based on G-learning - a probabilistic extension of the Q-learning method of reinforcement learning. In this paper, we demonstrate how G-learning, when applied to a quadratic reward and Gaussian reference policy, gives an entropy-regulated Linear Quadratic Regulator (LQR).


texts: Dividing data set into (almost) equal batches based on characteristics

texts:
I have a large dataset containing 6 characteristics (all numerical).
I need to split this dataset into multiple batches to be processed in parallel, and ideally, the batches should be as equal in size as possible.
The catch is: I can only split based on the 6 characteristics, so I have to specify ranges of values for each batch.
Simplified example: [1,1,1,2,2,3,3,3,3,4] will be split in two by specifying numbers 1-2 go to batch 1 and 3-4 go to batch two and I end up with [1,1,1,2,2] and [3,3,3,3,4].
I managed to make a simple division algorithm based on one characteristic, unfortunately some of the characteristics might not be suitable to split on, and I have no way to reliably predict which characteristic will create a good split.
I remember from Uni that a classification algorithm might do the trick here, but I am not sure how to implement the requirement for similar size classes.
here is the simple code I wrote for one characteristic, it may help to understand the situation:
def divide_into_groups(lst: list, num_groups=2):
    length = len(lst)
    sorted_list = sorted(lst)
    optimal_group_size = length // num_groups

    initial_group_markers = [sorted_list[i * optimal_group_size] for i in range(1, num_groups)]
    marker_edges = [(sorted_list.index(marker), length - 1 - sorted_list[::-1].index(marker))
                    for marker in initial_group_markers]

    final_group_markers = []
    for i, marker in enumerate(initial_group_markers):
        if i - marker_edges[i][0] < marker_edges[i][1] - i:
            final_group_markers.append(marker - 1)
        else:
            final_group_markers.append(marker)

    distances = [abs(initial_group_markers[i] - final_group_markers[i]) for i in range(len(initial_group_markers))]
    efficiency = (length - sum(distances)) / length
    print('distances from optimals: %s' % distances)
    print('efficiency of division is %s %%' % (efficiency * 100))

    return final_group_markers

P.S. this is my first question in this forum. If I am missing some information or have made any mistake, please comment and I will fix as soon as possible.


texts: Time Series Classification with multiple rows per date

texts:
I have a time series data set with the lifecycle of 9000 different B2B sales leads. What I call lifecycle consists of a dataset with one registry per day for every different sales Lead identifier with 4 predictive variables (DAYS_SINCE_START, LEAD_ID, CUSTOMER_INTEREST, MARKET, TYPE_SERVICE) and one response variable (OUTCOME). The response variable outcome can have 2 different values: Won (1) or Lost (0).

A mock example of the data frame would be the following:

As it can be seen, some leads “die” before others, this is because we receive the final outcome of the customer at that day of its lifecycle (we won or lost the lead), so that lead identifier drops from the dataset.

My mission is to create one single model that could be able to define the outcome of a new sales lead that is entering to the data set, and the prediction would be done 30 days after current time (Why 30 days after? The first 30 days after the resources of the company have already been assigned). How would I model this?


texts: What are bias and variance in machine learning?

texts:
I am studying machine learning, and I have encountered the concept of bias and variance. I am a university student and in the slides of my professor, the bias is defined as:
$bias = E[error_s(h)]-error_d(h)$
where $h$ is the hypotesis and $error_s(h)$ is the sample error and $error_d(h)$ is the true error. In particular, it says that we have bias when the training set and the test set are not independent.
After reading this, I was try to get a little deepr in the concept, so I searched on internet and found this video , where it defines the bias as the impossibility to capture the true relationship by a machine learning momdel.
I don't understand, are the two definition equal or the two type of bias are different?
together with this, I am also studying the concept of variance, and in the slides of my professor it is said that if I consider two different samples from the sample error may vary even if the model is unbiased, but in the video I posted it says that the variance is the difference in fits between training set and test set.
Also in this case the definitions are different, why?


texts: How to deal with a binary classification problem, where the instances in the negative class are very similar?

texts:
Let's say, one wants to detect, whether a picture of a fixed size contains a cat or not. But as a dataset, you have 10000 pictures of cats, and 30000 pictures which don't contain a cat, but are very similar to each other. For example, let's assume, the 30000 pictures in the "not cat" class contain only pictures of one or two kinds of spiders.
When training a CNN, you will find that you achieve a high score on the test set (here high score = almost fully diagonal confusion matrix) but when you want to use the CNN in the real world, you find that almost everything gets classified as a cat.
Why does the network generalize badly in this case? Even if the dataset doesn't represent the kind of data, the CNN would see in the real world, shouldn't it be easy for the CNN to say "I have seen 10000 examples of cats, therefore anything which doesn't look like a cat is not a cat" ?
How would one deal with this problem (besides gathering more data)?


texts:
I want to create a deep learning model for a binary classification problem. I have 16 features and 50k samples. How would I select number of layers and number of neurons of each layer  for my problem?

Please share the links which explains the same.


texts: Can 2 dimensional input be applied to SVM?

texts:
When considering Support Vector Machine, in an take in multiple inputs. Can each of these inputs be a vector??
What i am trying to say is, can the input be a 2 dimensional vector??


texts: counter vector fit transform cosine similarity memory error

texts:
count_matrix = count.fit_transform(off_data3['bag_of_words'])

I have count_matrix shape with

  count_matrix.shape
  (476147, 482824)

cosine_sim = cosine_similarity(count_matrix, count_matrix)

I think the matrix size is too big to cause this memory error

  --------------------------------------------------------------------------- MemoryError                               Traceback (most recent call
  last)  in

  ~/venv/lib/python3.6/site-packages/sklearn/metrics/pairwise.py in
  cosine_similarity(X, Y, dense_output)    1034     1035     K =
  safe_sparse_dot(X_normalized, Y_normalized.T,
  -> 1036                         dense_output=dense_output)    1037     1038     return K

  ~/venv/lib/python3.6/site-packages/sklearn/utils/extmath.py in
  safe_sparse_dot(a, b, dense_output)
      135     """
      136     if sparse.issparse(a) or sparse.issparse(b):
  --> 137         ret = a * b
      138         if dense_output and hasattr(ret, "toarray"):
      139             ret = ret.toarray()

  ~/venv/lib/python3.6/site-packages/scipy/sparse/base.py in
  mul(self, other)
      479             if self.shape[1] != other.shape[0]:
      480                 raise ValueError('dimension mismatch')
  --> 481             return self._mul_sparse_matrix(other)
      482
      483         # If it's a list or whatever, treat it like a matrix

  ~/venv/lib/python3.6/site-packages/scipy/sparse/compressed.py in
  _mul_sparse_matrix(self, other)
      514                                     maxval=nnz)
      515         indptr = np.asarray(indptr, dtype=idx_dtype)
  --> 516         indices = np.empty(nnz, dtype=idx_dtype)
      517         data = np.empty(nnz, dtype=upcast(self.dtype, other.dtype))
      518

  MemoryError:

Any tips to avoid this memory error when You have large matrix?


texts: Are there cases where tree based algorithms can do better than neural networks?

texts:
I trained an image auto-encoder on a large dataset, and now have for every image, an n-dimensional feature vector. This vector is not spatially correlated to the image. I now used this embedding space directly for classification, using fully connected dense layers. I have 55 classes, and the vector dimensionality is 1024. Even a single classification layer on top of the embedding, means 1024*55 parameters and it very easily overfits.

Since my embedding is not spatial in any sense, does it make sense to use Trees for classifying over the embedding space? I.e, feed the feature vectors to a tree based classification method, instead of a fully connected network. Are there cases, depending on the data, where trees outperform neural networks. If yes, what are such cases?


texts:
I am in an ML course and one of our tasks is to predict the helpfulness of Amazon reviews.  Currently, I am doing what most people seem to .  That is, a hashvectorizer(2-gram) on the text, tfidf, several scalar features, and (in my case) a Ridge classifier (parameters chosen by grid search).
  Honestly, it has been frustrating since I have tried to read and apply best practice from the course and any published works I can find.  That said, I am still only achieving marginal results:

{'Pos': 6588, 'Neg': 84412, 'TP': 2755, 'TN': 78990, 'FP': 5422, 'FN': 3833, 'Accuracy': 0.8982967032967033, 'Precision': 0.33692063103827807, 'Recall': 0.41818457802064357}

...which isn't great. I don't need help coding*, but if anyone can suggest useful case studies, or other sklearn classifiers/techniques I could research, I would be grateful.  I'm rapidly running out of directions to follow up on.

Thanks!

if relevant, you can find my latest iteration here


texts:
Based on the deeplearningbook:

$$MSE = E[(\theta_m^{-} - \theta)^2]$$

$$equals$$

$$Bias(\theta_m^{-})^2 + Var(\theta_m^{-})$$

where m is the number of samples in training set, $\theta$ is the actual parameter in the training set and $\theta_m^{-}$ is the estimated parameter.

I can't get to the second equation. Further, I don't understand how the first expression is gained.

Note:

$Bias(\theta_m^{-})^2 = E(\theta_m^{-2}) - \theta^2$

Also how bias and variance evaluated in classification.?


texts: LSTM loss value not change, accuracy stucked at 50%

texts:
I'm using LSTM for time series prediction, my data is highly skewed, with class weight 197.16865807 : 0.50127117

With Label 0 : 25359 and Label 1 : 9974641

my model is shown below

n_input = 100
n_features = 36
class_weights = class_weight.compute_class_weight('balanced',
                                              np.unique(y_target),
                                              y_target)
model = tf.keras.Sequential([
        tf.keras.layers.LSTM(64, activation='tanh', input_shape=(n_input, n_features),return_sequences = True),
        tf.keras.layers.LSTM(64, activation='tanh',return_sequences = True),
        tf.keras.layers.LSTM(64, activation='tanh',return_sequences = True),
        tf.keras.layers.LSTM(64, activation='tanh',return_sequences = True),
        tf.keras.layers.Dense(128, activation='relu'),
        tf.keras.layers.Dense(1,activation='sigmoid')])

model.compile(optimizer='adam', loss= 'binary_crossentropy' ,metrics=METRICS)
model.fit_generator(train_generator, epochs= 1,steps_per_epoch=len(train_generator),class_weight=class_weight)

I have tried the following method to dealing with my unchanged accuracy and loss value.

I tried to adjust class weight, but it seems not working, my predictions are all 1
I tried to use RandomSampler from imblearn to undersample my data, but the accuracy stucked at 50%
I tried to change the loss function to  weighted_cross_entropy_with_logits, but I did not find any examples show how to use it in Sequential model like the one above

I feel my model is not predict the result, since when I feed balanced dataset, the accuracy is around 50%, when I feed imbalanced dataset, the accuracy is 99%.

Can anyone help me with this? I wondering if it's the problem of my model, or the problem of my imbalanced dataset

Thank you!


texts: Weka Decision Tree not working on real data

texts:
I'm trying to make human activity recognition using the iOS accelerometer and gyroscope. The feature that I use are mean, variance, standard deviation, energy and cross-domain entropy. If I provide Weka with a test set of, let's say, 30% of the total data, it gives me an accuracy of 98.7%.
But when I implement the J48 decision tree in my application, the decision tree literally doesn't work and it doesn't classify activities correctly (the precision is near 0%), although I'm taking samples the same way that I take samples for the test set. The same data used for classification in the app gives incorrect results in the app but a 98% precision in weka. How is this possibile? Maybe I'm implementing the decision tree the wrong way?

I simply read the decision tree this way: if Var(Gravity.z) is less or equal than 47572.90768, then I check Energy(GravityMag) and if it's less or equal than 31250.004174, then I check Energy(UserAcc.y) and so on... since Weka assumes that if the difference between two numbers is smaller than 10e-6, I multiplied all numbers for one million so the precision goes up to 10e-12, and I double checked that I'm also correctly multiplying the values to be used by the in-app classifier so that couldn't be the problem. If the precision on the test set is high it couldn't be overfitting so I really don't know what to think.


texts:
I stuck on this topic for couple days and seems I need your help to understand what is expected value in TreeExplainer when feature_perturbation = tree_path_dependent, precisely what is expected value in this code?

I use SHAP 0.35, xgboost.

explainer = shap.TreeExplainer(model=model, feature_perturbation='tree_path_dependent', model_output='raw')
expected_value = explainer.expected_value

I know that if I use feature_perturbation = interventional then expected_value is just mean log odds from predictions:

explainer = shap.TreeExplainer(model=model, feature_perturbation='interventional', data= background_data)

#almost true
explainer.expected_value == model.predict(background_data, output_margin=True).mean()

Can you tell me what is it and how is obtained expected_value in the first code snippet?


texts: Anomaly Detection for Large Time Series Data

texts:
I am working on detecting anomalies within a large time series data set. It is updated on a regular basis and consists of more than 30 parameters. I am using R as a reference language.

It is a first for me working on this type of projects and I am unfamiliar with most of the techniques. I have 6 weeks to implement a
good analytical toolbox to enhance the quality of the control checks on the production line.

I have found a couple of potential methods to analyze it including statistical machine learning, deep learning using auto-encoded neural networks or clustering approaches. The purpose of the chosen method is to detect the anomalies/outliers by itself. It doesn't really need to be real-time analysis.
What approach would you recommend to implement for the scope of the project, given the structure of the data?


texts: Where to find Facial Visual Object Class

texts:
I am trying to make a face/head detection model using YOLO and darkflow.
In order to do this I need a set of images peoples face annotated in Visual Object Class, VOC format. I would prefer data set with face from lots of different angles and I am looking to train the model on about 1000~3000 data sets.
I've been doing some googling to find if there are any good data but have been failing to find any. I wanted to ask if someone knew a good source of them before I head on a journey to label over 1000 images.

For those who are looking to do the same.I have found some good pre-trained models.
1. YOLO-Face-detection
2. YoloKerasFaceDetection

I found 1. a little bit inaccurate for my project, it detects fine with threshold of about 0.1 but I believe it can still be improved with more data.
The method 2 is trained to do more than just detect faces.


texts:
I have downloaded Auto WEKA into my computer

and installed Java Runtime Environment

and I have WIndows 10

I am trying to run the Auto WEKA by double clicking on autoweka.jar file after assign jar files to Java but that does not work

I also tried using creating .bat file same as this link suggested
yet nothing works

https://stackoverflow.com/questions/394616/running-jar-file-on-windows

how can I get the GUI running?


texts: How to use testing data set to measure recommender system algorithm

texts:
I am new to recommender systems and am trying to build one using item-to-time CF. Currently, I am trying to evaluate/measure results using MAE. I have one step which is unclear (after I managed to split training dataset and testing data set and computed item-to-item similarities matrix). I understand that I can now do predictions for a given item for a specific user. Let's say from the testing data set, there is User A with 10 items already rated. What should I do with that user? Randomly pick one item I and pretend I didn't know the score of User A given to item I. Then calculate the predicted score using a weighted sum of rest of the 9 items?

Or, should I actually enumerate all 10 items that User A has ratings for. Then each time take out only the target item and using rest of the 9 items to do predictions?


texts: What are best ways to perform sentimental analysis on Twitter Data which I dont have labels for?


texts: Fetching rules from rpart using caret package

texts:
I'm looking to find a way where I can extract rules from my decision tree. I have 8 predictors and they are all categorical variables and the response variable has three outputs A B and C.

I've developed the model and can also create decision tree but I'm having trouble in understanding the tree and be able to extract rules from my model.

The 8 predictors I have can have values a, b, c, d, e.

To give you an example, I want to extract rules from my decision tree like this:

Rule 1:

Predictor 1 - a
Predictor 2 - b
Predictor 3 - c
Predictor 4 - b
Predictor 5 - a
Predictor 6 - d
Predictor 7 - e
Predictor 8 - a

Result: A

Rule 2:

Predictor 1 - a
Predictor 2 - b
Predictor 3 - d
Predictor 4 - b
Predictor 5 - a
Predictor 6 - d
Predictor 7 - b
Predictor 8 - a

Result: A

Rule 3:

Predictor 1 - a
Predictor 2 - b
Predictor 3 - d
Predictor 4 - a
Predictor 5 - a
Predictor 6 - a
Predictor 7 - b
Predictor 8 - a

Result: B

and so on and so forth.

Is there a way I have rules from my decision tree like I explained above?

Here is the result of my model:

124 samples
8 predictors
3 classes: 'A', 'B', 'C'

No pre-processing.
Resampling: Cross-Validated (10 fold, repeated 10 times).
Summary of sample sizes: 112, 111, 111, 112, 110, 112, ...
Resampling results across tuning parameters:

  cp           Accuracy   Kappa
  0.00000000  1.0000000  1.0000000
  0.06329114  1.0000000  1.0000000
  0.12658228  1.0000000  1.0000000
  0.18987342  1.0000000  1.0000000
  0.25316456  1.0000000  1.0000000
  0.31645570  1.0000000  1.0000000
  0.37974684  1.0000000  1.0000000
  0.44303797  0.7261846  0.5696935

Accuracy was used to select the optimal model using the largest value.
The final value used for the model was cp = 0.3797468.
n= 124

node), split, n, loss, yval, (yprob)
      * denotes terminal node

1) root 124 79 A (0.3629032 0.2741935 0.3629032)
  2) Activity4e< 0.5 45  0 A (1.0000000 0.0000000 0.0000000) *
    3) Activity4e>=0.5 79 34 C (0.0000000 0.4303797 0.5696203)
      6) Activity7c>=0.5 34  0 B (0.0000000 1.0000000 0.0000000) *
      7) Activity7c< 0.5 45  0 C (0.0000000 0.0000000 1.0000000) *


texts: Random Forest Overfitting, issues with mtry=1?

texts:
I am constructing what is known as an 'Expected Goals' model for football. This metric measures shot quality and a probability is assigned to a shot to achieve this, i.e. the chance a shot will be converted. To create this model I am using a random forest classifier. For evaluation purposes I am only interested in the accuracy of the probabilities rather than strictly classifying shots, therefore, I use the predictions to calculate the Mean Square Error where goal = 1 and no goal = 0. The MSE for the test set along with two benchmarks are as follows:

[1] "test.rf_mse: 0.0856633533734135"
[1] "comparison_model_mse: 0.0820007160001345"
[1] "naive_baseline_mse: 0.0912291249164997"

Note that the comparison model should be better than mine but the naive baseline should be worse. When looking at this the model looks to be doing okay. However, when I apply the same steps to the training set I get the following:

[1] "test.rf_mse: 0.0112001023587005"
[1] "comparison_model_mse: 0.0722459417565357"
[1] "naive_baseline_mse: 0.0858344459279039"

Here the MSE falls to unrealistic levels. Doesn't this mean that my model is overfitting? I understand that the idea that random forests can't overfit isn't strictly correct as all models can overfit to some extent, but for the model to be overfitting by this much must mean I am misunderstanding something here.

#FINAL MODEL FOR SHOTS DATASET
set.seed(5555)
trainIndex <- createDataPartition(shots$goal.miss, p = .75, list = FALSE)
train_set <- shots[ trainIndex,]
test_set  <- shots[-trainIndex,]

set.seed(1000)
rf.shots <- randomForest(as.factor(goal.miss) ~ ., data=train_set, ntree=500, mtry=5)

###TEST RESULTS
pred <- predict(rf.shots, test_set, type="prob")
test_set$predictions.test <- pred[,2]
mean(((test_set$goal.miss - test_set$predictions.test)^2))

###TRAIN RESULTS
pred <- predict(rf.shots, train_set, type="prob")
train_set$predictions.train <- pred[,2]
mean(((train_set$goal.miss - train_set$predictions.train)^2))

NOTE: I moved on to using the 'ranger' random forest package which allowed me to tune more hyperparameters. From a grid search, I found the only parameter that appeared to cause overfitting was mtry. I could remove overfitting by setting mtry to 1. However, having mtry at one does not minimise the MSE for the test_set and I have 29 features in the model meaning the default mtry should be 5. Therefore, I believe I must have some issues with my feature selections that is causing the overfitting. Either that or there are circumstances whereby mtry-1 is optimal? But I am not convinced by that.


texts: Visualizing a large graph (10'000 nodes)

texts:
I am currently trying to find a way to visualize a graph which has 10'000 nodes, and about 20'000 edges.
I was trying this javascript library, but quickly realised that this is too computationally intensive.
What are some viable options to visualize a graph with about 10'000 nodes, and up to 100'000 edges?

If there is no library that allows the computational effort, what are some techniques which will still allow me to gain valuable insight / or represent the graph without changing too much on it's structure?


texts: How can I get the predict future following value using Tensorflow LSTM?

texts:
Thank you for reading. I'm not good at English.

I am wondering how to predict and get future time series data after model training. I would like to get the values after N steps.

So, I used the time series in the Tensorflow tutorial to practice predicting the model.

a = y_val[-look_back:]
for i in range(N-step prediction): #predict a new value n times.
    tmp = model.predict(a.reshape(-1, look_back, num_feature)) #predicted value
    a = a[1:] #remove first
    a = np.append(a, tmp) #insert predicted value

The results are predicted by linear graphs from nonlinear graphs as shown below.

How can I get the actual, right solution? or Is Tessorflow impossible to predict multiple steps?

full source  (After the 25th line is my code.)

epoch=100


texts: How do I multiple individual time-series data to train a LSTM model?

texts:
I have 100 univariate time-series from individual patients measuring glucose levels. Each time-series is ~20k. I need to train my LSTM model using all of the data.

I don't think concatenating all into one single long time-series makes sense since all are individually collected independent of each other.

Can someone please help.


texts: How to evaluate performance of a time series model?

texts:
I trained a LSTM network on a time series dataset. Predictions seem to follow the dataset. In fact, they are nearly a right shifted form of real values. Thus, in my opinion, it doesn't provide any predictive capability since only shifting a signal in time results in low RMSE but is not useful.

How to properly evaluate a time series model?


texts:
I am trying to do a multivariate linear regression and I am having some issues. Namely, I am getting the following error:

ValueError: Cannot feed value of shape (3,) for Tensor 'X:0', which has shape '(1, 3)'

I have 3 feature variables, which I call trainX and 1 label, which I call trainY. Their shapes are the following (they are numpy arrays):

trainX.shape:
(2500, 3)
trainY.shape:
(2500,)

The following piece of code defines the tensors that I use to compute the model:

X = tf.compat.v1.placeholder("float", [1, 3], name="X")
Y = tf.compat.v1.placeholder("float", [1], name="Y")

W = tf.Variable(tf.zeros([3, 1]), name="W")
b = tf.Variable(tf.zeros([1]), name="b")

I calculate the predicted label and the cost function and the optimizer by doing:

predicted_y = tf.matmul(X, W) + b
cost = tf.reduce_sum(tf.pow(predicted_y-Y, 2)) / (2 * n)
optimizer = tf.compat.v1.train.GradientDescentOptimizer(learning_rate).minimize(cost)

I am getting the error in the tensor-flow session, namely in the following piece of code:

with tf.Session() as sess:
    sess.run(init)
    for epoch in range(training_epochs):
        for (_x, _y) in zip(trainX, trainY):
            sess.run(optimizer, feed_dict={X: _x, Y: _y})
        if (epoch + 1) % 100 == 0:
            c = sess.run(cost, feed_dict={X: trainX, Y: trainY})
            print("Epoch", (epoch + 1), ": cost =", c, "W =", sess.run(W), "b =", sess.run(b))
    # Storing necessary values to be used outside the Session
    training_cost = sess.run(cost, feed_dict={X: trainX, Y: trainY})
    weight = sess.run(W)
    bias = sess.run(b)

Any help would be greatly appreciated.


texts: human level performance on ImageNet, top-1 or top-5?

texts:
Anyone have pointers to where the human level performance on ImageNet comes from?

I found a reference to 5.1% accuracy (top-1? or top-5?) from here.


texts: Can GLM( generalized linear method) handle the collinearity between the predictor variables in a regression-analysis?

texts:
I'm a beginner in Machine learning and I've studied that collinearity among the predictor variables of a model is a huge problem since it can lead to unpredictable model behaviour and a large error. But, are there some models (say GLM) that are perhaps 'okay' with collinearity unlike the classic linear regression? It is said that the classic linear regression assumes there is no correlation between its independent variables.
This question arises because I was doing a project that said "If the input features are correlated with each other, it is better to use a generalized linear model since they would perform better than linear regression."
Can someone explain this ?


texts: Multiclass classification problem with multiple targets to be predicted

texts:
We have a supervised multi-class classification problem where we need to predict two targets for each sample: 'brand' and 'category'. Our features are 'shop_name' which can be any proper noun and several categorical features ('shop_type', 'building_type').
In the training data, 'brand' is not always present, but 'category' is always present. When brand is present, the category is directly dependent on it. For e.g. If the brand is 'Snickers', the category is already known to be 'Foodstuff'. In other cases, the category has to be determined from the input features.
Here is a representative example of our training data:

shop_name
shop_type
building_type
brand
category

Snickers Ltd

huge
Snickers
Foodstuff

Acme Intl
office stationary
medium

Office Equipment

Davidoff cigars
cigarette
big
Davidoff
Smoking

Sample Company
car repair
small

Automobile

Features: shop_name (free text), shop_type (categorical - 100 types), building_type (categorical - 20 types)
Target: brand (categorical - 2500 brands), category (categorical - 250 categories)
How should we structure our classification algorithm to determine brand and category? We have a well known list of about 2500 brands that we are interested in.


texts: How do Data Scientists integrate Predictive Modeling with SQL?

texts:
I've heard a lot about the importance of knowing SQL but how do you guys at the larger firms integrate your software for modeling/analysis with the server that holds the data?

Do you simply access some SQL Workbench and download a .csv file then read the file in your software? Or do you gain access to the server directly through your software?

This question is coming from the guy who learned SQL but never learned how to synchronize that with R.


texts: Calculation of relative standard deviation with a custom function in R


texts: What are the advantages and disadvantages of SQL versus NoSQL in data science?

texts: In which situations would one system be preferred over the other? What are the relative advantages and disadvantages of relational databases versus non-relational databases?


texts: Why is oversampling outperforming class weight?

texts: I have a dataset that is highly imbalanced. One class has 412 (class 0) samples while the other has 67215 (class 1) samples. For its classification, I am using MLP. When I use class weight of 165 for class 0 and 1 for class 1 (in keras), I am getting extremely bad results. However, if I oversample the dataset, I am getting really good results. What is the reason behind this?


texts: A question on realizable sample complexity

texts:
I came across the following exercise, and I just can't seem to crack it:

Let $l$ be some loss function such that $l \leq 1$. Let $H$ be some hypothesis class, and let $A$ be a learning algorithm. show that:
$m^{\text{stat, r}}_H (\epsilon) = O\left(m^{\text{stat, r}}_H (\epsilon/2, 1/2)\cdot \log(1/\epsilon) + \frac{\log(1/\epsilon)}{\epsilon^2}\right)$

Where $m^{\text{stat, r}}_H (\epsilon)$ is the minimal number $m$ such that for any realizable distribution over training examples $D$ we have that:
$$\mathbb{E}_{S \sim D^m}\left[ l_D(A(S)) \right]\leq \epsilon$$
And where $m^{\text{stat, r}}_H (\epsilon, \delta)$ is the minimal number $m$ such that for any realizable distribution over training examples $D$ we have that:
$$P_{S \sim D^m}\left( l_D(A(S)) \geq \epsilon \right) \leq \delta$$
Thanks a lot in advance!


texts: Parse documents to obtain subjective sentiment

texts:
I'm working on a project which deals with MRC (Machine Reading Comprehension).
I would like a machine to read an article and give me the sentiments based on a provided token.

For Instance given the input:  "Jackson Heights should see an increase in real estate, While other areas of brookln should see a steep decline".

Given the token: "Jackson Heights" Should return a score indicating positive sentiment: e.g .72

Given the token: "Brooklyn" Should return a score indicating negative sentiment: e.g -.72

I've tried to solve this problem in several different ways.  I've been following along with a tutorial for sentiment analysis.  I've linked to the corresponding jupyter notebook.
The issue is that this "Sentiment Analysis" doesn't really do much for multiple reasons.

Sentiment analysis removes stop words such as weren't and were.   So the statement "I would buy google stock" and the statement "I wouldn't buy google stock" would evaluate to the same sentiment
Sentiment analysis is only per text so understanding what subject applies to sentiment is important.  e.g "I love chicken and I hate fish" may return a negative sentiment because neural nets have a tendency to put a stronger weight on statements closer to the end.   However the sentiments to be tokenized based on the subject.

I've investigate a few open source technologies for this such as AllenNLP However, this didn't seem promising because the model assumes the question can be answered.
Given the text: "The price of GOOG should increase by 3% next week"
and a question "How much will Yahoo increase"
the output still highlights "3%" instead of returning nothing.
After I investigated  a simple Question answering bot using Kaggle.
This seems a bit more robust but it seems a bit overkill, I don't want to investigate a complicated AI model if there is a simpler solution.
To put the question in its simplest form.
I would like to parse a document into a list of tokens with their sentiment score.  However the solutions i've tried, remove important information, and full NLP seems to be a bit overkill.
could someone outline an simple efficient process for obtaining tokenized sentiment analysis?


texts: Test for feature dependencies in time series modelling

texts:
I have time-series data that track event occurrence in 3 locations. Here's a sample:
               Count     Total
Location       A  B  C
Date
2018-06-22     0  1  1     2
2018-06-23     2  1  0     3
2018-06-24     0  0  1     1
2018-06-25     2  2  1     5
2018-06-26     0  3  1     4

I would like to use the data to predict the total number of event occurrences at a given date in the future. How do I test if an event happening in one location has an impact on events happening in another location (dependency)? I believe that if an event happening in locations B and C are dependant, I should sum the 2 columns together as 1 feature in my model.


texts: I want to train a CNN. However, my input is images of size 100*100 with complex numbers. I have runned the model, but it failed and the loss didn't decrease. Then I found out that my because my inputs are complex, they are not able to train very good. Actually, I think the activation functions and the convolutions and the optimizer, backpropagation, cost should be changed. I wonder how I can Implement my CNN with Complex-Valued input. I would be thankful if anyone could help me.


texts: Feature selection by involving validation dataset

texts:
I need expert advice about a small algorithm created to perform features selection. I have used a genetic algorithm to perform features selection based on a specific objective function (good accuracy & sensitivity). in fact, I have involved the metrics resulting from testing the model on the validation dataset, during this process of feature selection for my objective function. here is simply the steps :
1- select random features (population)
2- perform training on the training dataset,
3- compute metrics using cross-validation on training dataset + compute metrics for that model on the 4- 4- validation dataset --> the combination is my objective function to maximize
5- save performance and features list
6- select best individu..
7- mutate ..
...
repeat
My question is according to machine learning guideline and proper practics am I allowed to use the validation dataset during the feature selection stage?
Thank you in advance
Additional info: I'm dealing with a small dataset, making features & parameter selection on the training dataset only always result in overfitting models that perform so bad on validation dataset.
Any suggestions and recommendations that could help will be appreciated.
thank you


texts:
I'm having a having a hard time understanding the difference between an isomorphism in graphs and canonical graphs. I have read through the Wikipedia articles, but it still isn't clicking.

Can somebody explain the difference, perhaps with an example?


texts: Who wrote the formula for gini importance/sklearn's feature importance score?

texts:
I've been looking for a paper where the Gini importance was first proposed, but I am not sure if this is actually how it came to be.

Here's the formula I am familiar with and am looking to find in a paper:

$$\frac{N_s}{N_t} * \left(i - \frac{{N_s}_r}{N_s} i_r - \frac{{N_s}_l}{N_s} i_l \right)$$

where

$N_s$ = number of samples at a particular node

$N_t$ = number of total samples

$i$ = Impurity

$*_r$ = measure of the right child node

$*_l$ = measure of the left child node

If I understand properly, this is the formula sklearn's random forests also use in model.feature_importances_

Tags (since I don't have the rep to create new ones yet): Feature Importance Score, Gini Importance


texts: Clustering of variants of similar news articles

texts:
We have data of several news sites, having quite literally millions of entries. As each news site publishes their own version of the news (also each news site may publish several different version of the same news), we have several entries that are variants of a single news. I am currently working on separating out "Unique" news from our repository. That means if a single news has several variants, only a single variant (most likely the one reported earliest) will be considered.

I believe, clustering of the news articles can be used to group together similar news. I am currently exploring DBSCAN, and Hierarchical clustering (Ward's Method). I am wondering whether am I moving in the right direction, is clustering the best solution for solving our problem? If yes, which other algorithms and techniques should I explore?


texts: How to choose the good number dimension of autoencoder?

texts:
I'm using Autoencoder for feature extracting. I stuck with how to choose good number of dimension of encoder layer (latent layer). After training dataset, the model gave the latent layer (embedding layer) with some zero value in the vector result.
For example, the embedding layer have 4 dimensions, one of node (unit) in embedding layer has value [0.67 0.0 2.13 0.43]. That I suppose they should 4 values different zero value.
I think my problem that I choose too many dimension for embedding layer which is not necessary and should smaller dimension, such as 3 or 2 dimensions.
So, my question how to choose good size for embedding layer?


texts: pytorch __init__() got an unexpected keyword argument 'train'

texts:
I'm using this code in colab. the function below gives me an error when intanciar.
import os
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
import torch.utils.data as data
import torchvision
from torchvision import transforms
def load_images(image_size=32, batch_size=64, root=""):
        transform = transforms.Compose([
        transforms.Resize(32),
        transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

        train_set = torchvision.datasets.ImageFolder(root=root,  train=True,transform=transform)
        train_loader = torch.utils.data.DataLoader(train_set, batch_size=batch_size, shuffle=True, num_workers=2)

        return train_loader

data = load_images (28.64, "/ content / data /")

error
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-85-a809f71788e0> in <module>()
----> 1 datotes=load_images(28,64,"/content/datitos/")

<ipython-input-84-0b4989f589a9> in load_images(image_size, batch_size, root)
     15     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
     16
---> 17     train_set = torchvision.datasets.ImageFolder(root=root,  train=True,transform=transform)
     18     train_loader = torch.utils.data.DataLoader(train_set, batch_size=batch_size, shuffle=True, num_workers=2)
     19

TypeError: __init__() got an unexpected keyword argument 'train'


texts: Fast checking of set membership on large data

texts:
Let's say that I am an email provider like Gmail. Let's assume that I have two categories of email-address: spammers and non-spammers. When my servers receive a mail I need to quickly check if that email-ID is in the spammers set and if yes I take some action.

The problem is that each email-Id could be several bytes ( say 10 bytes each) and I might have 1 billion spammers so I need 10GB of RAM to just store the emails in main memory.

Let's say I want to use only 1 GB RAM. To do this I'm now ready to accept an approximate answer. In particular I'm ok with a non-spam email mistakenly flagged as belonging to the spam  set but not vice versa. How will I do it?


texts:
I am an intern at mobility data company and a Master's candidate in Statistics. I am researching about driving score which is based on a driver's driving habit.
We have trip data which contains the distance, quick acceleration(seconds), quick stop, steering angles, and so on.
I have read some related articles and papers but some of them contain the skills that I cannot handle such as genetic programming.

I would like to know which ML skills could be used for this unsupervised learning problem(maybe?). Below is an example of our data set.

Driver ID | Trip time | Distance | Harsh acceleration | Quick Stop | ...

1              60 mins     1 mile       180 seconds         7 times  ...

2             30 mins      0.3 mile    10 seconds           2 times ...

My goal is making a driving score based on the dataset for each driver. The scale does not matter. It could be 0-100 scale or classifiers such as poor, bad, normal, good, perfect.
The problem I'm struggling with is that I have to create the target value(driving score). I guess that unsupervised learning could be a hint but I am not pretty sure about it.
I welcome any kind of advice or source! Thank you so much in advance.


texts: How to go about text mining for suggestions/Tips in reviews for restaurants?

texts:
For example for restaurants reviews usually have suggestions like "Go in the evenings", "order the so and so sauce with this dish"  or even "TIP: ask for the blah blah blah"

How can I detect such sentences? How do people usually tackle similar challenges?

Do they create classification rules like <modal_verb><preference_verb><optional_window_size_of_3><positive_sentiment_words>
Some examples of these rules are “would be great” and “could be really good”  --
 found this from here.

I guess I would have to use a tagger to categorize words?

Any blog that has attempted something similar step by step?

Any help would appreciated.


texts: dimensionality reduction of (partly) categorical metadata

texts:
I have several datasets, each with with hundreds of samples. I have different metadata for each data set, which contains about 50 variables per sample. Some of this metadata is clearly redundant. For example, each sample has multiple unique Id's related to different databases which I can easily remove/merge.
However, I know that other variables are also redundant. For example plot naming/numbering (it's agricultural data) overlaps with some tested variables. Is there an easy way to reduce the dimensions of my metadata? Much, but not all of it is categorical.
I was thinking that it might make sense to use Canonical Correspondence Analysis (CCA) here (similar to PCA).
I'm using julia for my project, but maybe I should do this part in r. Or does somebody know a package that can do CCA in julia?


texts: xgboost: give more importance to recent samples

texts: Is there a way to add more importance to points which are more recent when analyzing data with xgboost?


texts: Why are Neural Network predictions "correct", but offset from true value? Not using any past lagged values

texts:
I recently asked a similar question, but didn't get a response that really addressed/fixed the issue. Additionally, I've done some more work since then. I'm sorry for the long question below, I just want to make it very clear what I've done so far and why I'm so perplexed.
I am working with a neural network with the Keras package in R, trying to predict hourly Bitcoin price, 24 hours ahead.
Here is the code for my model:
batch_size = 2

model <- keras_model_sequential()
model%>%
  layer_dense(units=13,
             batch_input_shape = c(batch_size, 1, 13), use_bias = TRUE) %>%
  layer_dense(units=17, batch_input_shape = c(batch_size, 1, 13)) %>%
  layer_dense(units=1)
model %>% compile(
  loss = 'mean_squared_error',
  optimizer = optimizer_adam(lr= 0.000025, decay = 0.0000015),
  metrics = c('mean_squared_error')
)
summary(model)

Epochs <- 25
for (i in 1:Epochs){
  print(i)
  model %>% fit(x_train, y_train, epochs=1, batch_size=batch_size, verbose=1, shuffle=TRUE)
  #model %>% reset_states()
}

You may notice that I am working on a time-series problem, but not using LSTM. This is because none of my inputs are time-series values. They are all exogenous variables. You'll also notice that I commented out the line "model %>% reset_states()". I'm not sure if that's the right thing to do here, but from what I read, that line is for LSTM models and since I'm not using one anymore, I commented it out.
Again, because I have no time-series inputs, I also set "shuffle=" to TRUE.
So, below are the predictions in blue vs. true value in red:

You can see that the predictions very often (but not always) lag behind the true value. Additionally, this lag is not constant. Let me again emphasize that all of the variables are exogenous. There is not past-price input that the model could be using in order to generate these late predictions. AND don't forget I set shuffle= TRUE, which confuses me even further as to how the model could be giving such results if there's no way (that I know of) that it can "see" past values in order to replicate them.
Here is the graph of the training data fit:

It's harder to tell, but the lag exists in the training data as well. I'll also say that if I change how far ahead I'm trying to predict, the apparent prediction lag changes too. If I try to predict 0 hours ahead (so I'm "predicting" current price given current conditions), there is no lag in prediction.
I checked to make sure the tables/columns were set up right so that the model is trained on "current" conditions predicting price 24 hours ahead. I've also played around with the network architecture and batch size. The only thing that seems to affect this lag is how far ahead I'm trying to predict - that's to say, how many rows I shifted my Bitcoin Price column by so that "future" price rows are matched with past predictor rows.
Something else weird I've noticed is that with this non-LSTM model, it has a rather high error when training (MSE=0.07) which it reaches after only 5-9 Epochs and then doesn't go any lower. I don't think this is relevant because the LSTM model I used before achieved MSE=0.005 and still had the same lag issue, but I figured I'd mention it.
Any advice, tips or links would be enormously appreciated. I can't for the life of me figure out what's going on.


texts: cor() function giving unexpected result for numeric data frame(Updated)

texts:
I have a file named "specdata" which has a lot of CSV that each have the columns: ("Date", "sulfate", "nitrate", "ID"). I want to calculate the correlation between nitrate and sulfate, and I am using the cor() function from the stats package for this. The expected result looks different from mine and I don't know why.
If anyone has passed the R Programming course on Coursera, the question is from the programming assignment from week 2
setwd(dirname(rstudioapi::getActiveDocumentContext()$path))
corr <- function(directory, threshold=0){
  csvFiles <-  getFilesById(directory)
  dataFrames <-  readMultipleCsvFiles(csvFiles)
  noNADataFrames <- lapply(dataFrames, na.omit)
  acceptableDataFrames <- filterMoreThanThreshold(noNADataFrames, threshold)
  correlation(acceptableDataFrames)
}
getFilesById <- function(directory = getwd()){
  allFiles = list.files(directory)
  file.path(directory, allFiles)
}
readMultipleCsvFiles <- function(csvFiles) {
  lapply(csvFiles, read.csv)
}
filterMoreThanThreshold <- function(noNADataFrames, threshold){
  Filter(function(x){NROW(x)>threshold}, noNADataFrames)
}
bindMultipleDataFrames <- function(dataFrames) {
  Reduce(function(x, y) rbind(x, y), dataFrames)
}
correlation <- function(acceptableDataFrames){
  sulfate <- lapply(acceptableDataFrames, function(x){x[2]})
  sulfate <- bindMultipleDataFrames(sulfate)
  nitrate <- lapply(acceptableDataFrames, function(x){x[3]})
  nitrate <- bindMultipleDataFrames(nitrate)
  sulnit <-  cbind(sulfate, nitrate)
  cor(sulnit)
}
cr <- corr("specdata", 150)
head(cr)
summary(cr)

I'm supposed to get this:
 > head(cr)
 [1] -0.01895754 -0.14051254 -0.04389737 -0.06815956 -0.12350667 -0.07588814
     Min.  1st Qu.   Median     Mean  3rd Qu.     Max.
 > summary(cr)
 -0.21057 -0.04999  0.09463  0.12525  0.26844  0.76313

But instead I get this:
> head(cr)
           sulfate    nitrate
sulfate 1.00000000 0.06088486
nitrate 0.06088486 1.00000000
> summary(cr)
    sulfate           nitrate
 Min.   :0.06088   Min.   :0.06088
 1st Qu.:0.29566   1st Qu.:0.29566
 Median :0.53044   Median :0.53044
 Mean   :0.53044   Mean   :0.53044
 3rd Qu.:0.76522   3rd Qu.:0.76522
 Max.   :1.00000   Max.   :1.00000


texts: What is the learning path for the role of NLP engineer a beginner should follow?

texts: I am data science enthusiast and have interest in NLP. How should I develop my understanding for the domain and prepare for the role of an NLP engineer? What are the must have skills to master NLP? Suggest few good reading resources/online courses


texts: Supervised learning on sources of information with different importance

texts:
I am trying to classify customer support sessions using supervised machine learning.

In each customer support session I have 3 bags of information.
1. The title of the customer's complaint
2. Information about the device the customer was using
3. Text of the chat session with the customer support agent

In each customer support session, there are 6 different classes. Is it better to:
1. Train a classifier on each bag of information and have them vote on which class the session belongs to?
2. Put all of the information into a single set of features, and train a single classifier to determine which class the session belongs to?
3. Other?


texts: How to measure model success in production

texts: I have a model running on a productive system. The model predicts if some lead will become a sale. How would you develop a check, which checks the success and the accuracy of the model? There is a time component because it took months where the lead could become a sale.


texts: What does updated alpha mean in LDA model?

texts:
I'm trying to understand LDA model by reading through implementations of the algorithm.
Many implementations update alpha during training iterations with codes like:
psi_sum1 = 0.0e0;
psi_sum2 = 0.0e0;

for (int d = 0; d < doc_size; d++)
{
    psi_sum1 += digamma(float(n_dk[d][k]) + alpha_old[k]) - digamma(alpha[k]);
    psi_sum2 += digamma(float(doc_words[d]) + alpha_sum) - digamma(alpha_sum);
}

alpha[k] = alpha_old[k] * psi_sum1 / psi_sum2;

e.g. for 10 topics, the init alpha could be [1, 1, ..., 1] (length = 10),
after training, the alpha is updated to [0.29, 0.30, ..., 0.28] (length = 10).
What does this updated alpha mean?
Can I use the updated alpha to determine whether the topic number is appropriate?


texts:
There is a problem we are trying to solve where we want to do semantic search on our set of data,
i.e we have a domain specific data (example: sentences talking about automobiles)

Our data is just a bunch of sentences and what we want is to give a phrase and get back the sentences which are:

Similar to that phrase
Has a part of sentence that is similar to the phrase
Sentence which is having contextually similar meanings

Let me try giving you an example suppose I search for the phrase "Buying Experience", I should get the sentences like:

I never thought car buying could take less than 30 minutes to sign and buy.

I found a car that i liked and the purchase process was straightforward and easy

I absolutely hated going car shopping, but today i’m glad i did

I want to lay emphasis on the fact that we are looking for contextual similarity and not just a brute force word search.

If the sentence uses different words then also it should be able to find it.

Things that we have already tried:

Open Semantic Search (https://www.opensemanticsearch.org/) the problem we faced here is generating ontology from the data we have, or
for that sake searching for available ontology from different domains of our interest.
Elastic Search(BM25 + Vectors(tf-idf)), we tried this where it gave a few sentences but precision was not that great. The accuracy was bad
as well. We tried against a human curated dataset, it was able to get around 10% of the sentences only.
We tried different embeddings like the once mentioned in https://github.com/UKPLab/sentence-transformers and also went through the example
https://github.com/UKPLab/sentence-transformers/blob/master/examples/application_semantic_search.py and tried evaluating against our human curated set
and that also had a very low accuracy.
We tried ELMO(https://towardsdatascience.com/elmo-contextual-language-embedding-335de2268604) this was better but still lower accuracy than we expected and there is a
cognitive load to decide the cosine value below which we shouldn't consider the sentences. This even apply to point 3.

Any help will be appreciated. Thanks a lot for the help in advance


texts: Terminology: SOMs, batch learning, online learning, and stochastic gradient descent

texts:
I'm not sure which word to use to differentiate a self-organizing map (SOM) training procedure in which updates for the entire data set are aggregated before they are applied to the network from a training procedure in which the network is updated with each data point individually.

In the case of other algorithms I'd say stochastic gradient descent, but I'm not sure gradient descent is correct for SOM learning.  To my knowledge, SOM learning does not follow any energy function exactly, so that should mean it doesn't do gradient descent exactly, right?

Another term would be online learning, but `online' sort of implies I train my SOM in the real world with data points streaming in, eg. from a set of sensors.  It may also imply that I use every data point only once, which is not what I want to say.


texts: VC dimension of hypothesis space of finite union of intervals

texts:
I have the following concept:
$$C = \left\{\bigcup_{i=1}^{k}(a_i, b_i): a_i, b_i \in {\Bbb R}, a_i < b_i, i=1,2,..,k\right\}
$$
and was wondering how to determine the VC dimension of C?


texts: count_vectorizer.vocabulary_.items() and count_vectorizer.vocabulary_ Inconsistent number of returns

texts:
# create a count vectorizer object
count_vectorizer = CountVectorizer()

# fit the count vectorizer using the text data
count_vectorizer.fit(data['text'])

# collect the vocabulary items used in the vectorizer
dictionary = count_vectorizer.vocabulary_.items()

To my understanding, after count_vectorizer fits to data['text'], it generates a list of features. In my case, it generated 25,257 features and these are mapped as dict data type when I call count_vectorizer.vocabulary_. Which is still 25,257 tuples. It means, it used all the features.

Problem is, when I call count_vectorizer.vocabulary_.items() it returns 15,142 tuples as  dict_items. Why the number has been reduced here? Should't all the features be used to make the dictionary?

Here are the lengths I'm talking about:

len(data['text'])  #19579
len(count_vectorizer.get_feature_names())  #25257 items
len(count_vectorizer.vocabulary_)  #25257 items
len(dictionary) #15142 items (??????)


texts: Language Model with Attention not learning

texts:
Language model with attention layer is not learning after 20 epochs. Both training and validation loss increase together, while the accuracy flattens at around 7%
The way input data is pipelined is by applying window of length 10 for each sentence so that the model will be able to make inferences on all words of vocab size rather than masking words randomly in each sentence. You can find the code here https://drive.google.com/file/d/1La83LKaZNHsGfCtxKtAWgAqAzLwHo9U5/view?usp=sharing

Orange = train
Blue = validation
Any suggestions on how to get this working would be helpful. Thanks.


texts: Connecting Infusionsoft data to Google data studio

texts:
I want to create a Google Data studio dashboard from Infusion soft data.
The main problem are the connectors - there are multiple tools that provide direct connectors but they are paid solutions like Klipfolio, Clicdata, Grow etc.

If a direct connection is not possible, I want to use some combination of Google sheets and Zapier or other free tools to create a data flow that can be constantly refreshed for data coming in from "infusionsoft" to "Google data studio"

Questions summary:

Is there any direct way of connecting Infusionsoft data or API with Google data studio (maybe with some Database like MySQL/postgresql/cloud sql etc?)
What are some free mid-tier tools I can use to create this data connection?(Zapier, google sheets, excel etc)

All your suggestions are highly appreciated.


texts: ValueError trying to use a pickled scikit-learn model

texts:
I am new to data science and trying to learn something. I was able to complete the prediction with 98% accuracy and i saved it as pickle model. Now while trying to predict using this model I am getting the below error.

trainFile=os.path.join('D:\PYPrograms','Data','POS','collected.csv')
#load the data
train  = pd.read_csv(trainFile)
dataTemp=train
nullInTrain=train.shape[0] - train.dropna().shape[0]
print("Null values in Train data "+str(nullInTrain))
dataTemp.columns = dataTemp.columns.str.strip().str.lower().str.replace(' ', '_').str.replace('(', '').str.replace(')', '')
dataTemp.loc[:,"title"] = dataTemp.title.apply(lambda x : " ".join(re.findall('[\w]+',x)))
df1 = dataTemp.dropna()
cv1 = CountVectorizer()
df_x = df1["tickettype"]+" "+df1["title"]
df_y = df1["type"]
X_train, X_test, y_train, y_test = train_test_split(df_x, df_y, test_size=0.2, random_state=0)
x_traincv = cv1.fit_transform(X_train)
x_testcv = cv1.transform(X_test)
clf = RandomForestClassifier(n_estimators = 1000, max_depth = 6)
clf.fit(x_traincv,y_train)
pred=clf.predict(x_testcv)
pred
#make prediction and check model's accuracy
predictions_test = clf.predict(x_testcv)
acc =  accuracy_score(np.array(y_test),predictions_test)
print ('The accuracy of Random Forest is {}'.format(acc))
import pickle
modelFile=os.path.join('D:\PYPrograms','Data','model2')
with open(modelFile, 'wb') as picklefile:
    pickle.dump(clf,picklefile)

with open(modelFile, 'rb') as training_model:
    model = pickle.load(training_model)

cv2 = CountVectorizer()
File=os.path.join('D:\PYPrograms','Data','POS','Report_one_wk08.csv')
data = pd.read_csv(File)
data.columns = dataTemp.columns.str.strip().str.lower().str.replace(' ', '_').str.replace('(', '').str.replace(')', '')
test = cv2.fit_transform(data['title'])
model.predict(test)

Error

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-205-c0ac8462bce6> in <module>
----> 1 model.predict(test)

~\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\ensemble\forest.py in predict(self, X)
    543             The predicted classes.
    544         """
--> 545         proba = self.predict_proba(X)
    546
    547         if self.n_outputs_ == 1:

~\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\ensemble\forest.py in predict_proba(self, X)
    586         check_is_fitted(self, 'estimators_')
    587         # Check data
--> 588         X = self._validate_X_predict(X)
    589
    590         # Assign chunk of trees to jobs

~\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\ensemble\forest.py in _validate_X_predict(self, X)
    357                                  "call `fit` before exploiting the model.")
    358
--> 359         return self.estimators_[0]._validate_X_predict(X, check_input=True)
    360
    361     @property

~\AppData\Local\Programs\Python\Python37\lib\site-packages\sklearn\tree\tree.py in _validate_X_predict(self, X, check_input)
    400                              "match the input. Model n_features is %s and "
    401                              "input n_features is %s "
--> 402                              % (self.n_features_, n_features))
    403
    404         return X

ValueError: Number of features of the model must match the input. Model n_features is 6639 and input n_features is 3

Data available at https://drive.google.com/open?id=1xaKKSXzpr7THezqU_8jycfvAueg0nnCQ


texts: ANN applied to Boston Housing dataset returns negative value

texts:
This example is taken from the book Deep Learning With Python from Jason Brownlee. It  applies a fully connected neural model with one hidden layer  (13, 13, 1) using Keras library and the Tensorflow backend.

1 - Import the packages

import numpy
from keras.models import Sequential
from keras.layers import Dense
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.datasets import load_boston

2 - Load the dataset

boston = load_boston()
X = boston.data[:,0:13]
Y = boston.target

3 - Define base model

def baseline_model():
    # create model
    model = Sequential()
    model.add(Dense(13, input_dim=13, \
                    kernel_initializer='normal', activation='relu'))
    model.add(Dense(1, kernel_initializer='normal'))
    # Compile model
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model

4 - Fix random seed for reproducibility

seed = 7
numpy.random.seed(seed)

5 - Fit & evaluate model

estimator = KerasRegressor(build_fn=baseline_model, epochs=10, \
                           batch_size=5, verbose=0)
kfold = KFold(n_splits=10, random_state=seed)
results = cross_val_score(estimator, X, Y, cv=kfold)

6 - Value of results parameter

Out[63]:
array([-13.89524042, -14.2215869 ,  -6.21156782, -42.65242339,
       -26.58890147, -56.30860755, -28.6575911 , -89.67339525,
       -27.7172946 , -22.67604859])

Which returns a mean negative value

print("Baseline: %.2f (%.2f) MSE" % (results.mean(), results.std()))

It returns about -30 and it should return about +30 according to the book. I've tried both Theano and Tensorflow with no success.

I've also tested this code both on Windows and Linux, having obtained the same bad result.

The problem seems to be in the cross-validation part, because if I don't run cross validation, I get more sensible results.

7 - Without cross-validation

model = Sequential()
model.add(Dense(13, input_dim=13, kernel_initializer='normal', \
                activation='relu'))
model.add(Dense(1, kernel_initializer='normal'))
model.compile(loss='mean_squared_error', optimizer='adam')
model.fit(X, Y, epochs=150, batch_size=10)

Now if I evaluate the model I get a more sensible value

In [68]: model.evaluate(X, Y)
506/506 [==============================] - 0s 58us/step
Out[68]: 27.778296670423664

What can be happening here? Why is the cross-validation procedure returning negative values?


texts: Metrics/Methods for deciding duration of video retention for on-demand websites

texts:
This might be a general question but I thought this might be the best place to brainstorm. If I had a video website that only wants to retain videos on-demand for a certain number of days before getting rid of them to maximize operating dollars, what sort of data measures/methods would I want to consider or use in general to decide the time period?

Just like a typical website, the video can be tracked for views, followers etc. and the trade off could be on views/operating cost


texts: What is the purpose of setting an initial weight on deep learning model?

texts:
I'm now learning about deep learning with Keras, and to implement a deep learning model at Keras, you set the initializer to set its initial weights on.

from keras.models import Sequential
from keras.layers import Dense
model = Sequential()
model.add(Dense(12, input_dim=8, kernel_initializer="random_uniform"))

The kernel_initializer can take something others, such as random_normal, which uses Gaussian, not uniform distribution, and zero, which literally sets all weights to 0.

However, I don't understand why you like to set different weights at the initializer. Specifically, what advantages does it have over setting all initial weights to 0, which sounds more natural for novices like me?

Also, should the initial weights, if needed, be always set a tiny value (e.g. 0.05) to?


texts:
I’m performing PCA on different time series’ and then using K Means clustering to try and group together common factors.

The issue I’m facing is that some of the factors come in and out of the time series. For example I may have 12 years in total of data points, some factors may exist for the entire 12 years but some may dip in and out (active for the first two years, inactive for three years, active for the rest for example).

I can use iterative PCA to fill in some of the usual gaps in data but I am unsure of what to use for the large chunks (years) of data gaps. For example if a factor only exists for the final three years should I even incorporate it into the analysis?


texts:
I have a dataset which have both categorical features with high cardinality (>8000) and low cardinality (4 or 5).
Would that be ok to encode the high cardinality ones with one encoder (target encoder, for example) and the others with low cardinality with another encoder (one hot encoder) and put everything together to train a model?
Is this wrong and should apply the same encoder to all the features regardless of their cardinality?

Many thanks for your inputs!


texts:
How to define a custom performance metric in Keras?

I am trying to use it but I can not see the metrics values on each epoch.

clf51.compile(optimizer=sgd51, loss='binary_crossentropy', metrics=["accuracy"])

clf51.fit(X_train, Y_train, batch_size=384, epochs=5, callbacks=[metrics], validation_split=0.30, verbose=2)

As you can see:

Train on 139554 samples, validate on 59810 samples

Epoch 1/5 - 4s - loss: 0.3576 - acc: 0.9885 - val_loss: 0.0531 - val_acc: 0.9989

Epoch 2/5 - 3s - loss: 0.0261 - acc: 0.9987 - val_loss: 0.0135 - val_acc: 0.9987

Am I doing something wrong? I would like to make a earlystop using f1s.


texts: How many parameters does a single stacked LSTM have? The number of parameters imposes a lower bound on the number of training examples required and also influences the training time. Hence knowing the number of parameters is useful for training models using LSTMs.


texts: What is the minimum requirement for the dataset for time series forecasting?

texts:
I have a dataset of patients where, for each patient, a measurement is taken 3 times per day. For example, patient 1 has recordings at 7.30 am, 12.30 pm and 8.30 pm. Patient 1 has a collection of 30 days with such data recorded thrice a day.(altogether 3 recordings per day * 30 days = 90 data points).

Is this type of data suitable for time-series forecasting?

Thanks in advance for the help.