texts: How to modify a Convolutional Neural Network architecture built for a univariate time series to multivariate time series?

texts:
I have built a CNN (in combination with a LSTM cell) that takes 1D time series-like data as an input and performs classification. I am obtaining a good performance, but the complete data has actually 6 dimensions and I am just using one at a time (each one individually obtains very similar performances).
I tried using the same architecture with the 6D data, and it produces a very marginal improvement over using just 1D. I am not sure if this is because the architecture should be different to account for the additional dimensions, or it might be the case that there is actually no advantage to using the full 6 dimensions.
Is there any modification I should be applying to the architecture in order to improve performance?


texts: Using Boruta for determining the magnitude of importance of R, F and M

texts:
I am a novice to R and sorry if this question is missing the mark.

I am working on a dataset with high seasonality and trying to assign weights to recency, frequency, and monetary value. For this, I am using the Boruta library in R. Based on the weights of R, F and M I am computing clusters using K-means clustering

My data set is from 2017-19 transaction data.
I am isolating customers from a year ago and computing how much revenue they generated in 2019
From my data set of customers of 2018 and before who made a purchase in 2019, with 'how much they spent in 2019' as the target variable, I am using Boruta.
Basis the weights of R, F, and M I am multiplying my columns and computing K-mean clusters

Questions: 1. I understand Boruta is done when there are multiple features - can I use it to compute the weights of R, F, and M? Is there a better way?
2. Is this a good approach to assigning weights to R, F, and M and computing the clusters
3. Is it normal for Boruta to give different outputs every time I run it?
3. Should I ignore recency in my RFM analysis because Boruta is indicating that?


texts: Role derivative of sigmoid function in neural networks

texts:
I try to understand role of derivative of sigmoid function in neural networks.

First I plot sigmoid function, and derivative of all points from definition using python. What is the role of this derivative exactly?

import numpy as np
import matplotlib.pyplot as plt

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def derivative(x, step):
    return (sigmoid(x+step) - sigmoid(x)) / step

x = np.linspace(-10, 10, 1000)

y1 = sigmoid(x)
y2 = derivative(x, 0.0000000000001)

plt.plot(x, y1, label='sigmoid')
plt.plot(x, y2, label='derivative')
plt.legend(loc='upper left')
plt.show()


texts: How do I perform multi-step forecasting on LSTM trained with multiple observations of a sequence?

texts:
I have a trained sequence-to-sequence LSTM that predicts a person's velocity trace (Δx, Δy) given their position (x, y) as they walk along a path. I trained the LSTM using 200 examples of different people walking along the path. Now, I would like to use multi-step forecasting to predict the future velocity of several new people after they have walked 1/3 of the path. Is this possible? If so, how can I implement this in Python or MATLAB?

Train = 200 sequence observations, 2 input features (x, y), time lengths of observations vary

Test = 5 partial sequence observations (1/3 of the sequence), I would like to predict the last 2/3 of the sequence for each person.


texts: How to increase the weight when it comes to outlier detection

texts:
Let's say we have feature A, B, C, D, E to represent one observation in an outlier detection model. We are using scikit-learn outlier detection in our case.

AFAIK, if we normalize all the features, they are as equally important as others.

What if I want to make B very important, e.g. if A, C, D, E increase 20%, the observation might still be considered as normal case while B increases 10%, the observation needs to be marked as an outlier.

Is there any way to increase or decrease features' weight in outlier detection model?


texts:
A part of our system does a limited rule-based word labeling (kinda WSD). Basically it takes a dependency tree of a sentence, finds words or POS tags of interest and assigns to each a label/sense based on the word itself, it's POS tag, position in sentence and so forth.

I was wondering if ML methods in general and Tensorflow in particular can be used for the task, and if so which ones to try. I'm familiar with theoretical basics of ML, but never actually used it in work, so I'd grateful for any pointers.

Basically, our input is a ConLL representation of a sentence that we can turn into an matrix:

[
 ['cat', 'NN', 'nsubj']
 ['jumped', 'VBD', 'root']
 ['over', 'IN', 'case']
 ['fox', 'NN', 'nmod:over']
]

we can encode words and labels to numbers and pad sentence to max length of say 60 words, which gives the final input 60x3 matrix of integers. Our output then is a vector of 60 numbers with codes for labels for corresponding input words.

So far my research points to using bidirectional RNN for the task, but all tutorials I can find use them either for classification (as with MNIST) or finding the next best word as in the Tensorflow's RNN tutorial, whereas I want to label each word/input.

So, my questions are:

Is a bidirection RNN a good choice for the task? And if so, what should the composition be for it? How many steps and hidden layers would you recommend?
If RNN isn't a good choice, what else would you recommend?
Would using word embeddings add to the accuracy if our labeling doesn't care for word meanings/similarities?


texts: Random Parameters to fix in ML to perform controlled experiments

texts:
Many algorithms and methods in modern Machine Learning techniques contain randomness, and because of that, running the same ML script several times can result in different outputs, therefore accuracy results. For example, running Random Forest can produce accuracy of 0.78, then when run again with no change in data, setup, code, it can result in 0.79. This brings the challenge of inability to perform controlled experiment, when I am testing some changes in input and their effect on output.

So in order to be able to perform perfectly controlled experiments to achieve the best model output, what are the extensive set of random parameters I should fix? I want whole process to be completely deterministic.

PS: I am using Sci-kit Learn environment with additional algorithms such as XGBoost, CatBoost, LightGBM.

I assume there are some parameters, random_state(s) I should fix in NumPy, too.


texts: Derivative of a custom loss function with the logistic function

texts:
I have costum loss function with $\mu ,p, o, u, v$ as variables and $\sigma$ is the logistic function.
I need to derive this loss function.
Due to multiple variables in the loss function, I need to use the softmax function which is the generalization of the logistic function?
$L = -\frac{1}{N}\sum_{i,j \in S}^{2}{a_j\{y_{i,j}log[\sigma{(\mu + p_i + o_j + u^{T}_{i}v_{j})]} + (1 - y_{i,j})log[1 - \sigma{(\mu + p_i + o_j + u^{T}_{i}v_{j})]}\}}$
As far I understand, it is a multivariate gradient descent problem so I need to provide 4 different update rules (doing partial derivative for each of the variables as the others are constants) but how I "merge" them into one update rule or maybe there is only one update rule from the beginning, I got pretty confused when it's become multivariable problem.


texts: Agent Collapse / Overfitting during Training

texts:
I'm new to reinforcement learning so please bear with me. I'm training an agent to play ms-Pacman using the actor-critic method.
Below are the results of a couple of runs, in both graphs the orange line in the average of the previous 100 values. The left graph is the episode duration and the right is the reward earnt in each episode.
In both the runs, the agent steadily learns and then collapses, and is unable to recover. Has the agent reached it's maximum potential with this method or is this a case of overfitting?
Hyperparameters for both the runs:

Batch Size: 128
Replay Memory Buffer: 500,000
Epsilon Minimum: 0.1
Epsilon Maximum: 1.0
Epsilon minimum is reached at around 2,000,000 steps (approx episode 3,000)


texts: Binary encoding and its interpretation in Python

texts:
I have a column named Street that has 2 values: Paved and Gravel.  Here is what print(train[binary_columns[0]].unique().tolist()) gives me:

['Pave', 'Grvl']

I want to encode these values in binary like this:

df['Street'] = df['Street'].replace(['Pave', 'Grvl'], [1, 0])

But I wonder if this is a good idea.  Wouldn't the computer interpret this as Pave > Grvl?  How does the computer differentiate between binary and integer encoding?


texts:
I have a script which I wrote using python and tflearn. I created a regression neural network model which takes in chemical analysis of wine as input and predicts a score out of 10.

Dataset: http://archive.ics.uci.edu/ml/datasets/Wine+Quality

The problem I have is that the prediction of my model is very bad. I'm also new to tflearn, I have only coded classification neural networks(twice). So, I'm a complete beginner in coding regression and in using tflearn.

Code:

import pandas as pd
import numpy as np
import tflearn
from tflearn.layers.core import input_data, fully_connected, dropout
from tflearn.layers.estimator import regression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt

def preprocess():

    data_source_red = 'F:\Gautam\...\winequality-red.csv'
    data_source_white = 'F:\Gautam\...\winequality-white.csv'

    data_red = pd.read_csv(data_source_red, index_col=False, sep=';')
    data_white = pd.read_csv(data_source_white, index_col=False, sep=';')

    data = pd.concat([data_red, data_white])
    data = data.dropna(inplace=False)

    x = data[data.columns[0:11]].values

    y = data[data.columns[11]].values

    sc = StandardScaler()
    x = sc.fit_transform(x)

    y = np.expand_dims(y, -1)

    x = np.float32(x)
    y = np.float32(y)

    return (x, y)

x, y = preprocess()

train_x, test_x, train_y, test_y = train_test_split(x, y, test_size = 0.2)

network = input_data(shape=[None, 11], name='Input_layer')

network = fully_connected(network, 5, activation='relu', name='Hidden_layer_1')

network = fully_connected(network, 1, activation='linear', name='Output_layer')

network = regression(network, batch_size=64, optimizer='sgd', learning_rate=0.2, loss='mean_square', metric='R2')

model = tflearn.DNN(network)

model.fit(train_x, train_y, show_metric=True, run_id='wine_regression', validation_set=0.1, n_epoch=10)

result = model.evaluate(test_x, test_y)
print('Accuracy is %0.2f%%' % (result[0] * 100))

pred_y = model.predict(test_x)

plt.plot(test_y, color = 'red', label = 'Real data')
plt.plot(pred_y, color = 'blue', label = 'Predicted data')
plt.title('Prediction')
plt.legend()
plt.show()

Again, the prediction is very bad. Also, the loss and R2 values go hectic. Sometimes, the loss is low(6.45) and sometimes very high(23445.45). The same goes for R2 value too. Sometimes, R2 value goes above 1.0

Even if the loss is minimum(0.1) and R2 as 0.95, the graph shows that the actual value varies lot from the predicted values.

What mistake am I doing? Why is my prediction very bad? And why are the values R2 and loss too high sometimes and too low sometimes?

Am I missing something here? This is my first regression neural network, so I don't know much about this. I hope that my question is clear. Thanks.


texts: I do not understand Generalized Regression. What is the purpose of the Neural Networks structure? And how does it works?


texts: How do I learn experimental methodology? When is it relevant?

texts:
I just graduated in Computer Science, with a very theoretical background but without any kind of Data Science or Artificial Intelligence experience, and I working on my own to discover those two fields. More precisely, I try to work on a toy subspace clustering k-means-like algorithm, and I think I successfully learned basic optimization techniques.

So now I have some ad-hoc subspace clustering algorithm, and... I'm stuck. How do I validate it? During my studies, I studied a lot of formal proving or model checking techniques, but I feel they are totally irrelevant here. I read several AI papers, and it seems that there is a strong tradition of experimental validation in Clustering, with a lot of validity measures. My problem is that they mean nothing to me. I don't understand what they prove, or even why they are relevant. I would be very interested in a general course about the experimental validation methodology (if it actually is a thing!), if possible with theoretical justification.

Moreover, I don't really know what are the good properties people are looking for in clustering algorithms. What general theorems should I aim for? I understand this may be a question too related to what I am working on - a generic answer for k-means-like algorithms would be enough.


texts: Can KMeans clustering be used on word2vec output?

texts: I have a dataset that has been trained on word2vec. Is it a good idea to cluster the output vectors?.


texts:
I'm trying to do multi-step regression and I use an output layer:

LSTM(1, activation='linear', return_sequences=True)

Is this the wrong way of achieving this?
Should I use a TimeDistributed(Dense(1)) as output? If yes, why?


texts:
I read several papers, where they propose to implement residual blocks of ResNet as follows
$$
u^{k+1} = u^k - \tau K^T \sigma(K u^k),
$$
where $u^{k}$ denotes output on k-th layer, $\tau$ is artificial time-step and K is convolution matrix.
The ResNet with this type of blocks should be more numerically stable as with more common block: $u^{k+1} = u^k - \tau K_2 \sigma(K_1 u^k)$.
How does one share weights and implement $K^T$ operation using keras/tf?


texts: Why did sampling boost the performance of my model?

texts:
I have an imbalanced dataset with 88 positive samples and 128575 negative samples. I was reluctant to over/undersample the data since it's a biological dataset and I didn't want to introduce synthetic data. I built a Random Forest Classifier with this original dataset. I got an F1 score of 0 for the positive class. Zero precision. Zero recall. I cross-checked the predictions and test data. The model predicts some positives none of which are actually positive. Worst performance.

So, I tried to oversample the positive class. I upsampled the positives to 1000 samples. To my surprise, the F1 score for this dataset was 0.97, for the positive class. Then I tried lesser samples. I was able to achieve an F1 score of 0.83 with 200 positive samples, which is just 2.25 times of the original positive samples.

I would like to know why this occurs. For 88 samples, F1 score is 0.00 (rounded off to two digits). For 200 samples it's 0.83. There is no data leakage. All the features are engineered. I used imbalanced-learn module for oversampling. Can someone explain why is this difference in performance?


texts: How to create a new dataframe using the another dataframe

texts:
I have created and worked on a DataFrame for a project. It looks like the following:

Critics   Items    Ratings

a...........1..........5

b...........2..........3

b...........3..........2

c...........8..........1

a...........1..........5

b...........4..........4

My DataFrame has 1M+ rows and 8 columns.

I want to create a new DataFrame where the rows are the unique critics, the columns are the unique items, and the individual cells are the rating a critic has given for the particular item. If the critic has not reviewed the item then I want to add an NA over there.

I tried doing the following for the rows:

ratings = pd.DataFrame(f.review_profilename.unique())

For the columns, I saw a lot of answers involving people using

ratings.rename(<individual column names>, axis='columns')

But this doesn't help me since I can't list down all the unique item names.

Edit:
I fixed the issues by using pivot tables. I am new to pandas and was not aware of something like this existing. The exact syntax I used was

ratings = f.pivot_table(index = 'critic',columns = 'item', values = 'ratings')


texts: Having averaged trials which are less than the number of features

texts:
Suppose I have an experiment where I have 70 features and 48 samples. The target variable is binary (0,1) and the 48 samples are divided such that 24 of them correspond to outcome 1 and the other 24 correspond to outcome 0.

Note that the data I have is an averaged data. In other words, each of the 48 samples is a result of "n" averaged trials.

I am interested in using SVM and obtaining the weight vector (70x1).
In a way, I have less number of samples vs features.
However, the data which I have is an outcome of averaged trials.

In case I don't have access to the raw data, what is the best way to deal with the data? And does having the mean of trials as my data give a privilege over having 48 raw-trails instead?


texts: Threshold logic unit (McCulloch-Pitts 1943) vs. Perceptron (Rosenblatt 1958)

texts:
I have searched various sources to find out what distinguishes the McCulloch-Pitts neuron from the perceptron invented by Rosenblatt. In most sources only one of these elements is considered, in others they are used as synonyms.
Can someone explain the differences in how they work?


texts: What is difference between Bayesian Networks and Belief Networks?

texts:
While reading some articles about Bayesian Networks, I came across many occurrences of Belief Networks.

Do both of these terms mean the same thing or is there any difference between Bayesian Networks and Belief Networks?


texts: Natural Language text categorization using RapidMiner

texts:
I'm new to data mining, so this might sound like a very simple task to some.

So, I work in reliability engineering in aviation & I have a set of data that is generated on a daily basis regarding system failures & failure rectification. This data is categorized using numerical tags of maintenance manual tasks (reference data) by chapter, section and paragraph. However, since the data is entered manually by people, sometimes, the wrong chapter/section tags entered & would require to be manually checked to insure the data's validity.

The failure/resolution data is available in table format (CSV, Excel...) and I also have the maintenance manual data keywords including their chapters/sections in table format.

My question is, is it possible using RapidMiner to take these tables, cross check some keywords in the text (failure/rectification) & compare it with the reference data, and output it with the proper reference tags (of the chapters, sections...), taking into account spelling errors, acronyms & abbreviations, or is there a program/application that is more specialized than RapidMiner to do these functions?

Example: Failure on system X was rectified and logged on the database. System X is under chapter 4 section 33, however, when entering the data, the person put it under chapter 3 section 44. I have a document that has the reference for system X being under Chapter 4 Section 33. Is it possible for RapidMiner to check the text in the failure text & the rectification text & cross-check it with a predefined list where System x is under Chapter 4 Section 33 & give me the output with the right Chapter/Section, taking into account spelling mistakes & the fact that some people write abbreviations and acronyms differently (example, I.B.M/IBM/I.B.M.)


texts: Given a sequence of inputs/outputs and a set of nodes that modify that input, can you find the topology of a graph?

texts:
I am working on a problem where I have to model a graph topology, where the nodes are logic/arithmetic operations that can be applied to the input. The network receives a multi-dimensional input, and returns a multi-dimensional processed output with less dimensions. The only things I have to do this are a set of input/output pairs and a set of probable nodes.

The system itself follows a few constraints:

The nodes are unique, so there is a limited amount of possible subsets that can compose the graph.
The input must enter a single node, and the output must be returned from a single node, even if the path of the information branches within the network.
In case multiple topologies are possible, the minimal is the answer.

There are also a few aggravants:

Since the nodes are a "probable list", there is no guarantee that the system is possible, but it is important to try to find the closest match possible, as a machine learning system uses the closeness to satisfaction to find new sets of probable nodes.
Nodes might receive multiple or just one input, and they might output one or multiple streams of output.

Given this system, I am wondering if anyone here knows of an existing algorithm that solves this problem(or something close that can be adapted).

My question thus is as follows:

Given a set of arithmetic and logical functions and a set of input/output pairs, is there an algorithm to find a graph of functions that satisfies all inputs/output pairs?


texts:
Is there an optimized way to perform this function "PolynomialFeatures" in R? I'm interested in creating a matrix of polynomial features i.e. interactions between two columns among all columns but I can't find a base function or a package that does this optimally in R and I don't want to import data from a Python script using sklearn's PolynomialFeatures function into R.

The idea is to transform this:

array([[0, 1],
       [2, 3],
       [4, 5]])

into this:

array([[ 1.,  0.,  1.,  0.,  0.,  1.],
       [ 1.,  2.,  3.,  4.,  6.,  9.],
       [ 1.,  4.,  5., 16., 20., 25.]])

Intercept and only-interaction options are well, optional. I could expand-omit those myself.


texts:
As the title states, I am trying to cluster a huge dataset and cluster it by using sklearn.Birch to learn incrementally.

If it's a small dataset, I could just use gridsearchcv.

However, there's no built-in way to do that with large dataset in scikit-learn right now.

I'm curious if there is any suitable/general way to tune parameters batch by batch?


texts: What is meant by frequency of a time series? I am having data observations of energy consumption from 01/05/2016 - 31/05/2017 such that there are 96 values per day. What will be the frequency of the time series in such case?


texts: Filtering detail table based on selection in summary table in Orange 3.11

texts:
I'm rather new to Orange, so apologies in advance if I'm missing something obvious or am trying to fit a square peg in a round hole.

I have two files, Summary and Detail, that I'm importing into an Orange workflow. Summary has one row per Contact, and Detail has multiple rows per Contact, each row corresponding to a particular action they've taken. Contacts are identified in both files by Contact ID.

I've done a bunch of analysis to identify Contacts in Summary that I'm interested in using Orange's machine learning algorithms. I'd like to be able to step through each row in Summary (that is, each Contact) to see what actions in Detail that the Contact took.

I thought I'd be able to use the Merge Data widget to do this by hooking up Selected Data from Summary as the Data and Data from Detail as the Extra Data using a Find Matching Rows join type. However, since each unique Contact ID appears multiple times in Detail, I'm not able to select it in the Merge Data widget.

I know it's nonsensical, but I did try to wire Data from Detail as the Data and Selected Data from Summary as the Extra Data with the only Merge Data option that worked (Append columns from Extra Data). This showed me all the data from Detail, but appended with columns from Summary when the appropriate row was selected in Summary. Based on my understanding of Merge Data, this is predicted but (in my use case) unhelpful behavior.

In database terms, I'm sort of trying to set up an inner join with a one-to-many relationship between Summary and Detail and, by selecting a row in Summary, to be able to see the related rows in Detail.

So, the bottom line is, is this sort of behavior doable in Orange, or am I trying to paint a house with a chainsaw? Is there another widget that would be more appropriate for the task? (I also tried the Select Rows widget, but couldn't figure out how to populate the filters based on the row that I had selected in Summary.)


texts:
I am trying to do clustering in my dataset which has 4 numerical fields. Please find the file attached : http://www.filedropper.com/example_3

I tried with this code:

from sklearn.cluster import KMeans
kmeans = KMeans(n_clusters=2, random_state=0, max_iter = 300).fit(dffinal)

I know that there are 2 classes in this example, that was the reason i tried with 2 clusters. Out of 4200 rows, first 3196 rows belong to a class and the remaining rows belong to an another class.

But when i do clustering, cluster labels are randomly assigned, and accuracy is less than 10%. Just wondering whether my features are not good enough for clustering or should I try with some other clustering algorithm.

Any help would be appreciated. Thanks.


texts: How to solve warning : ImageDataGenerator specifies `featurewise_center`, but it hasn't been fit on any training data?

texts:
I am doing image classification and I have a training set and a test set with different distributions. So to try to overcome this problem I am using an Image generator in the following way:

trainingset = '/content/drive/My Drive/Colab Notebooks/Train'
testset = '/content/drive/My Drive/Colab Notebooks/Test'

batch_size = 32
train_datagen = ImageDataGenerator(
    featurewise_center=True,
    featurewise_std_normalization=True,
    rescale = 1. / 255,\
    zoom_range=0.1,\
    rotation_range=10,\
    width_shift_range=0.1,\
    height_shift_range=0.1,\
    horizontal_flip=True,\
    vertical_flip=False)

train_generator = train_datagen.flow_from_directory(
    directory=trainingset,
    target_size=(256, 256),
    color_mode="rgb",
    batch_size=batch_size,
    class_mode="categorical",
    shuffle=True
)

test_datagen = ImageDataGenerator(
    featurewise_center=True,
    featurewise_std_normalization=True,
    rescale = 1. / 255
   )

test_generator = test_datagen.flow_from_directory(
    directory=testset,
    #target_size=(256, 256),
    color_mode="rgb",
    batch_size=batch_size,
    class_mode="categorical",
    shuffle=False
)

num_samples = train_generator.n
num_classes = train_generator.num_classes
input_shape = train_generator.image_shape

So I use featurewise_center=True and featurewise_std_normalization=True, which by doing some research I have found that it should solve the problem, at least a little bit.

But then if I build my CNN and train it, I have the following warning:

I don't understand how to solve this, since if I try to do the fitting as said in the warning, it says me that I need a numpy array, while I have a string:

train_datagen.fit(trainingset)

I get:

    ValueError                                Traceback (most recent call last)
<ipython-input-7-afd35f2bf342> in <module>()
     29     vertical_flip=False)
     30
---> 31 train_datagen.fit(trainingset)
     32
     33 train_generator = train_datagen.flow_from_directory(

1 frames
/usr/local/lib/python3.6/dist-packages/numpy/core/_asarray.py in asarray(a,
dtype, order)
     83
     84     """
---> 85     return array(a, dtype, copy=False, order=order)
     86
     87

ValueError: could not convert string to float: '/content/drive/My Drive/Colab
 Notebooks/Train'

I have been stucking in this for days, but I don't understand how to solve this problem. Can somebody pleas help me? Thanks in advance.


texts: Decision tree with final decision being a linear regression

texts:
Question:
I want to implement a decision tree with each leaf being a linear regression, does such a model exist (preferable in sklearn)?

Example case 1:

Mockup data is generated using the formula:

y = int(x) + x * 1.5

Which looks like:

I want to solve this using a decision tree where the final decision results in a linear formula. Something like:

0 <= x < 1 -> y = 0 + 1.5 * x
1 <= x < 2 -> y = 1 + 1.5 * x
2 <= x < 3 -> y = 2 + 1.5 * x
etc.

Which I figured could best be done using a decision tree. I've done some googling and I thought the DecisionTreeRegressor from sklearn.tree could work, but that results in points being assigned a constant value in a range, as shown below:

Code:

import matplotlib.pyplot as plt
import numpy as np
from sklearn.tree import DecisionTreeRegressor

x = np.linspace(0, 5, 100)
y = np.array([int(i) for i in x]) + x * 1.5

x_train = np.linspace(0, 5, 10)
y_train = np.array([int(i) for i in x_train]) + x_train * 1.5

clf = DecisionTreeRegressor()
clf.fit(x_train.reshape((len(x_train), 1)), y_train.reshape((len(x_train), 1)))

y_result = clf.predict(x.reshape(len(x), 1))
plt.plot(x, y, label='actual results')
plt.plot(x, y_result, label='model predicts')
plt.legend()
plt.show()

Example case 2:
Instead of one input, there are two inputs: x1 and x2, output is computed by:

x1 = 0 -> y = 1 * x2
x1 = 1 -> y = 3 * x2 + 5
x1 = 6 -> y = -1 * x2 -4
else -> y = x2 * 20 - 100

Code:

import matplotlib.pyplot as plt
import random

def get_y(x1, x2):
    if x1 == 0:
        return x2
    if x1 == 1:
        return 3 * x2 + 5
    if x1 == 6:
        return - x2 - 4
    return x2 * 20 - 100

X_0 = [(0, random.random()) for _ in range(100)]
x2_0 = [i[1] for i in X_0]
y_0 = [get_y(i[0], i[1]) for i in X_0]
X_1 = [(1, random.random()) for _ in range(100)]
x2_1 = [i[1] for i in X_1]
y_1 = [get_y(i[0], i[1]) for i in X_1]
X_2 = [(6, random.random()) for _ in range(100)]
x2_2 = [i[1] for i in X_2]
y_2 = [get_y(i[0], i[1]) for i in X_2]
X_3 = [(random.randint(10, 100), random.random()) for _ in range(100)]
x2_3 = [i[1] for i in X_3]
y_3 = [get_y(i[0], i[1]) for i in X_3]
plt.scatter(x2_0, y_0, label='x1 = 0')
plt.scatter(x2_1, y_1, label='x1 = 1')
plt.scatter(x2_2, y_2, label='x1 = 6')
plt.scatter(x2_3, y_3, label='x1 not 0, 1 or 6')
plt.grid()
plt.xlabel('x2')
plt.ylabel('y')
plt.legend()
plt.show()

So my question is: does a decision tree with each leaf being a linear regression, exist?


texts: Implementing single variable Linear Regression in python

texts:
I'm trying to implement LMS algorithm in python. I have the following code:

import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

def compute_cost(theta, X , y):
    inner = np.power(X.dot(theta) - y, 2)
    return np.sum(inner/2*len(X))

if __name__ == '__main__':
    path = os.getcwd() + '/data/ex1data1.txt'
    data = pd.read_csv(path, header=None, names=['Population', 'Profit'])
    data.plot(x='Population', y='Profit', kind='scatter', figsize=(10, 10))
    plt.show()
    # append a ones column to the front of the data set
    data.insert(0, 'Ones', 1)

    # set X (training data) and y (target variable)
    cols = data.shape[1]
    X = data.iloc[:, 0:cols - 1]
    y = data.iloc[:, cols - 1:cols]
    theta = np.array([0,0])
    print(X.shape, theta.shape, y.shape)
    print(compute_cost(theta, X, y))

The first print statement (printing X.shape, theta.shape, y.shape) prints the following:

(97, 2) (2,) (97, 1)

When I try to compute the cost function I get the following:

Profit    0.0
0         0.0
1         0.0
2         0.0
3         0.0
         ...
92        0.0
93        0.0
94        0.0
95        0.0
96        0.0
Length: 98, dtype: float64

However, I'm supposed to (according to the exercise) get 32.07
I think that the bug is related to the theta shape but I tried initializing theta like this:

theta = np.zeros(shape=(2, 1))

And still this doesn't work..
To be clear, I'm not looking for help writing the algorithm simply find the syntax bug, understand why it happened and fix it.


texts: How to deal with skewed imbalanced image dataset to work with CNN?

texts: I am working on multi-class classification problem on an image dataset. There is one class with 80% of the images and rest 20% is divided into rest 6 remaining classes. If I have to apply the image-augmentation technique, what ratio of count of images between each class I have to maintain?


texts: Classify Customers based on 2 features AND a Time series of events

texts:
I need help on what should be my next step in an algorithm I am designing.

Due to NDAs, I can't disclose much, but I'll try to be generic and understandable.

Basically, after several steps in the algorithms, I have this:

For each customer that I have, and events that they do during a month, during first steps I have clustered the events into several categories (each customer will have the events separated into categories that go from 1 to x being x between 1 to 25, generally the first categories have more density of events than the others).

For each category and customer I have created a time series aggregating the events of the month per hour (getting patterns of when these events are being done). Also I am using a couple of normalizing variables based on the number of days over a month (30 days) that the guy performs at least one event, and the number of days with at least one event over the total of days with at least one event (aggregating all clusters). The first one gives me a ratio of how active the customer is during the month, and the second one weights the category against the others.

The final table looks like this

|*Identifier*|  *firstCat* | *feature1* | *feature2*  |   {      *(TIME SERIES)*   }

CustomerID  |  ClusterID |  DaysOver30 | DaysOverTotal | Events9AM Events10AM ...

 xx | 1 | 0,69 |  0,72 |  0,2   0,13   ...

 xx | 2 | 0,11 |  0,28 |  0,1   0,45   ...

 xy | 1 | 0,23 |  0,88 |  0,00  0,60   ...

 xy | 2 | 0,11 |  0,08 |  1,00  0,00   ...

 xy | 3 | 0,10 |  0,04 |  0,40  0,60   ...

The time series variables are the percentage over the total of events per day on each specific category (this means that per each row adding up all variables should be 1). The reason of doing it like that is because for example a time series with events 0 0 0 1 0 and 1 1 1 2 1 are completely different, and standardizing to normal would give similar results. And due to high skew between different categories, I check the values on the time series independently with the others.

What I need to do now is to identify these categories (remember, they can be from 1 to x being x any number from 1 to 25) into 3 tags: tag A, tag B and None of Them. Looking at these variables I can manually identify which tag they belong to, and the idea is to identify manually as much as I can and use any classifier algorithm to learn from that and identify all of them.

My idea was to use multiple logistic regressions on the table, but all the variables of the time series are correlated (since they are a linear combination of each other), so I thought I better use a clustering algorithm only over the time series using euclidean distance to categorize the different patterns and use the result and the other two normalizing variables in the logistic regression.

The other concern that I have is that this approach takes each row independently from the others, and in theory, for each customer there should be only 0 or 1 tag A, 0 or 1 tag B and the remaining of them should be None (another tip is that normally Tag A and B are between first categories, because is highly dependent on the normalizing features (if days over total is High, there is a high possibility that the row is either A or B, depending on the Time Series Pattern).

Edit: This is no longer a concern, I will just perform two different logistic regressions, one for Tag A or Other and another for Tag B or another, with the result probabilities I can select only the best of each.

The dataset is enormous and the final algorithm needs to be applied using SQL (on Teradata), but for getting the coefficients of the logistic regression, or the centers of the clustering I get a sample and use R.


texts: Difference between $Q(s,a)$ ,$V^*(s)$ and $V^\pi(s)$ in Markov Decision Process?

texts:
I am new to RL and I am trying to understand how to find solutions of an MDP.
This is what I understand so far -> since the nature of our environment is stochastic, at a state 's' if we take an action 'a' we do not know which state we would end up in. So we define the following terms :

T(s, a, s') or P(s'/s, a) - transition probability. This is the probability that starting at state 's' and taking action 'a' we end up in state s'

V(s) - value of a state, which tells us how good or bad this state 's' is.

R(s) - reward of being in a state 's', this is also known as the immediate reward of being in state 's'

Q(s) - state action value function. This tells us that given we are at a state 's' and we take a particular action 'a' what is the expected utility of the state s' we might end up in

Now coming to the part that confuses me.
From what  I understand so far, the difference between v(s) and q(s) is that v(s) gives us the utility of a state assuming we do not know which action to take while q(s) is the value of that state given we take a particular action 'a'.

Coming to $v^\pi(s)$ = same as what I defined v(s) to be, so the value of a state
$v^*(s)$ = value of a state given we take the optimal action 'a' so this is like following max (q(s)) over all actions

I read online how to mathematically define these terms and I am not sure if I correctly understand them. Also, I don't really understand the difference between $v^*(s) $ and $v^\pi(s)$
I found some conflicting ideas online but since I am just starting out I am not sure what is lacking in my understanding:

$
V^\pi(s) = R(s) + \gamma*max\sum P(s'/s,a)*V^\pi(s')
$

$V^*(s) = max\ Q*(s,a)$ and here we can substitute for Q*(s,a) as $Q^*(s,a) = \sum T(s,a,s')*[R(s) + \gamma V^*(s')]$ so we get $V^*(s) = max  \sum T(s,a,s')*[R(s) + \gamma V^*(s')]$

$V(s) = \gamma*max\sum P(s'/s,a)*V(s')$

As you can see there are different equations which are defining the same V(s) quantity? So what exactly is the clear definition of V(s)? Are these equations just dependent on the case of RL we are dealing with?
I don't understand how to distinguish between these, any suggestions/links/readings are much appreciated! Thanks!


texts: Can we combine decision tree , xgboost , lightgbm to make one hybrid classifier?

texts: Can we combine three different classifier to make one hybrid classifier ?


texts: Use Python to Scape Data on website ends with aspx

texts:
https://blue.kingcounty.com/Assessor/eRealProperty/default.aspx

This is the website I want to scrape.

For example, I input 500 4TH AVE
I just want to get Parcel Number and Lot size.

I was trying to edit the link

https://blue.kingcounty.com/Assessor/eRealProperty/search=5004THAVE.aspx

I think it did not work, please demonstrate. Thank you.


texts: Classification models with multi-class allowed for each record

texts:
I am training a multi-class classification model. Each record can belong to one or more classes. (actually can I still call it a classification model? or should it be a topic model or something else?)

The training data looks like this (format_A)

record_id, f_1 , f_2, f_3, ... , f_n, output_A, output_B, output_C
        1,  0.1, 0.2, 0.3, ... , 1.2,        1,        0,        1
        2,  0.3, 1.2, 0.5, ... , 2.1,        1,        0,        0
        3,  0.2, 3.2, 1.3, ... , 0.4,        1,        1,        0
        4,  1.1, 0.1, 0.7, ... , 0.5,        0,        0,        1
        5,  2.1, 0.5, 0.8, ... , 1.9,        0,        1,        1
        6,  0.5, 0.4, 0.2, ... , 0.8,        1,        1,        1
                     :
                     :

Right now, I am using the sklearn logistic regression with multi-class. I preprocess the above training data such that there is only one target per record. For record with more than 1 output, I duplicated the records. That is, the above training data becomes below (format_B):

record_id, f_1 , f_2, f_3, ... , f_n,    target
        1,  0.1, 0.2, 0.3, ... , 1.2,  output_A
        1,  0.1, 0.2, 0.3, ... , 1.2,  output_C
        2,  0.3, 1.2, 0.5, ... , 2.1,  output_A
        3,  0.2, 3.2, 1.3, ... , 0.4,  output_A
        3,  0.2, 3.2, 1.3, ... , 0.4,  output_B
        4,  1.1, 0.1, 0.7, ... , 0.5,  output_C
        5,  2.1, 0.5, 0.8, ... , 1.9,  output_B
        5,  2.1, 0.5, 0.8, ... , 1.9,  output_C
        6,  0.5, 0.4, 0.2, ... , 0.8,  output_A
        6,  0.5, 0.4, 0.2, ... , 0.8,  output_B
        6,  0.5, 0.4, 0.2, ... , 0.8,  output_C
                         :
                         :

I use format_B to train a multi-class logistic regression model. The model then predicts the probability of each category, and I set my own threshold to decide the final outputs.

While I have done the above, I am wondering would it possible to use format_A as the input of sklearn logistic regression with multi-class directly?

Or is there any package/library actually takes format_A as input? Or any other kind of models actually more suitable for this type of problem?

Thank you!


texts: Is there a possibility that there is no relationship between some inputs and outputs?

texts:
I have a general question that comes to my mind, I'm doing machine learning projects and I took a look at many datasets and worked with, mostly there are already famous datasets that everyone uses.

Now the question comes to my mind, let's say I decided to make my own dataset, is there a possibility that my data are so random so that no relationship exist between my inputs and outputs ?? this is interesting because if this is possible, then no machine learning model will achieve to find an inputs outputs relationship in the data and will fail to solve the regression or classification problem.

Moreover, is it mathematically possible that some values have absolutely no relationship between them? in other words, there is no function (linear or nonlinear) that can map those inputs to the outputs.

Now I thought about this problem and concluded that if there is a possibility for this, then it will likely happen in regression because maybe the target outputs are in the same range and the same features values can correspond to the same output values and that will confuse the machine learning model.

What are your thoughts on this? did you had this problem in your daily life as a machine learning engineer, data scientist or hobbyist?


texts:
I need help reshaping a DataFrame that i got from a csv file.

In this file I have the first column as ID and then 97 columns that represent sale units of 4 products during time.

the data is like

id    p1_201501   p1_201502 ......  p1_201812  p2_201501..... p2_2018012   p3_..

x12     125         12        ......  06          500     ..... 14         2...
nz15     250         16        ......  600         423     ..... 312        56...
....

The Id represent the area in which the product is sold. And I have 700 rows (i.e 700 df['id'].nunique() ) And so I need to find the yearly sales trend of a given product. So i thought the solution is to change the dataframe in order to have years as rows and the quantity for each product (p1, p2, p3)

Something like:

id    date        p1   p2    p3
x12   201501     12   500   32
      201502    ....
      201503    ....
      ....
      201812    .....
nz15  201501    .....
      201502    .....
      .....
      201812    .....

But I don't know if it is the right choice to find the saled trend in this case and if so How to change the first dataframe to be like the second I know some about pivot reshape. But I can't find how to do it and how to keep the id (the area in which the product is sold)

Any help please?


texts: Preventing fitting Regression CNN to the mean when dataset has only few outliers

texts:
I am trying to train a CNN for regression on a dataset where most of the points lie around a similar output value. There are however a few outliers that are very important but they are less represented and the trained network thus tends to predict all output values close to the mean of the whole dataset (underfitting). This leads to a somwhat small error (and good precision) because the vast majority of points lie in that range, but the error is way higher for points even slightly outside the "normal" case.

But since this regressor would be most useful to predict the output for outliers (quality-control use case), it's currently pretty much useless.

Is there a way to prevent this kind of behavior and have a CNN that is giving a greater weight to outliers and extrema, in order to avoid underfitting?

To some extent, the Random Forest method, although much better at predicting the output for outliers, still exhibits a higher error for points in the extrema, while the error around the mean is very small. The "low" points are predicted with too high a value, and the "high" points are predicted with too low a value (each time closer to the mean). So any idea for that case would be great too!

Thanks a lot


texts: How to determine if a company decision was successful or not?

texts:
I'm trying to figure out if a decision taken in a company (offering discounts for specific products) is successful or not.
I have done some research and saw that A/B testing might be a way to do this but A/B tests can only be carried out having 2 groups (control and experiment), and in this case all I have is past and "after the decision" data.

Can the statistics behind AB testing be used in this case?

The data I have is in the for of sales per day per article only for one company before and after the decision (only one decision).

Example:

sales for articles X and Y from the 2016-01-01 to 2016-02-01.
The decision was to lower discounts on articles A and B on 2016-01-15, so I want to know if sales decreased or not after this decision.


texts: Improving the performace of the Naive Bayes classifier by decorrelating the data

texts:
I was wondering if it is possible to improve the performance of the Naïve Bayes classifier by decorrelating the data. The Naïve Bayes assumes conditional independence of the features given some class $P(a_1, a_2 | c_1) = P(a_1 | c_1)P(a_2 | c_1)$ which is not necessarily true.

What if we applied a transformation into the space using decorrelation which will result in $Cov(a_1, a_2) = 0$ for all features. We have that if $a_1$, $a_2$ independent, then $Cov(a_1, a_2) = 0$. Although the converse does not strictly follow (e.g. consider the two dependent random variables $X \sim Norm(0, 1)$, and $X^2$ whose covariance is 0), the new features will be closer to being independent.


texts: intuitive understanding of (1+parameter)/parameter

texts: When looking at a function that contains term (1+parameter)/parameter, should there be an intuitive understanding for why this term is included. For example if a function for fishing mortality includes this term and "parameter" is a a growth parameter for fish, is there something I can say about the function with regard to fish growth? Why is the value 1 added to parameter in the numerator?


texts: Can't save save/export and load a keras model that uses eager execution

texts:
I'm following the RNN text-generation tutorial with eager execution pretty much line for line. I've trained the model with my own data set and have saved a low loss checkpoint. I'm able to load the weights and generate text but I want to export/save the model so that I can learn how to deploy one using flask. However I can't figure out how. The version I'm using is '1.14.0-rc1'.

The tutorial: https://www.tensorflow.org/tutorials/sequences/text_generation

I have been able to save the model as an HDF5 file but I cannot load it. I've also disabled eager execution but that causes problems with running the code later on. I have tried the following and a few more snippets but those led to nothing as well:

new_model = keras.models.load_model("/content/gdrive/My Drive/ColabNotebooks/ckpt4/my_model.h5")

How ever I get

RuntimeError: tf.placeholder() is not compatible with eager execution.

Lastly I found this in another post and tried it as well but was met with another error:

tf.saved_model.save(model, "/content/gdrive/My Drive/Colab Notebooks/ckpt4/my_model.h5")

error:

AssertionError: Tried to export a function which references untracked object Tensor("StatefulPartitionedCall/args_2:0", shape=(), dtype=resource).TensorFlow objects (e.g. tf.Variable) captured by functions must be tracked by assigning them to an attribute of a tracked object or assigned to an attribute of the main object directly.


texts: Is window based sequencing a good idea to obtain more training data for LSTMs?

texts:
I am trying to do an unsupervised autoencoder based outlier detection for time series using LSTMs. Here, there are multiple time series, and an entire series is to be considered as an outlier.
However, I only have around 25-30 time series instances to work on (though each series comprises ~10k points).
I wanted to know whether creating sliding windows for each time series to generate more data is an okay idea for training the autoencoder more accurately. Also, in that case, how does one merge the results to selectively identify which of the original 25-30 time series is an outlier?
I am relatively new at working with LSTMs and would really appreciate suggestions on whether this idea is feasible.


texts: calculate distance between each data point of a cluster to their respective cluster centroids

texts:
I have a dataset of some keywords in some text files. Using the append feature I have access each text file and I append all of the keywords to token_dict like this

token_dict="wrist. overlapping. direction. receptacles. comprising. portion. adjacent. side. hand. receive. adapted. finger. comprising. thumb. ..............................."

By using k-means clustering, I clustered this data by using k=3. Now, I want to calculate the distance between each data point in a cluster to its respective cluster centroid. I have tried to calculate euclidean distance between each data point and centroid but somehow I am failed at it. My code is as follows:

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer, TfidfTransformer
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
from matplotlib import style
import numpy as np
style.use('ggplot')
token_dict = []
import glob
path = 'E:\\Project\\*.txt'
files=glob.glob(path)
for file in files:
    f=open(file, 'r')
    text = f.read()
    token_dict.append(text)
vectorizer = TfidfVectorizer(max_df=0.8, max_features=10000, min_df=2, use_idf=True)

#print(X)

km = KMeans(n_clusters=3)
#labels = km.fit_predict(vectorizer)
#print(labels)
X = vectorizer.fit_transform(token_dict).todense()
km.fit(X)
pca = PCA(n_components=2).fit(X)
data2D = pca.transform(X)
# =============================================================================
# cluster_0=np.where(X==0)
# print(cluster_0)
#
# X_cluster_0 = data2D[cluster_0]
# print (X_cluster_0)
# =============================================================================

# =============================================================================
# def euclidean(X1, X2):
#     return(X1-X2)
# =============================================================================

# =============================================================================
# distance = euclidean(X_cluster_0[0], km.cluster_centers_[0])
# print(distance)
# =============================================================================
# =============================================================================
#
# km.predict()
# =============================================================================
order_centroids = km.cluster_centers_
centers2D = pca.transform(order_centroids)
labels = km.labels_
colors = ["y.", "b.","g."]
for i in range(len(X)):
    plt.plot(data2D[i][0], data2D[i][1], colors[labels[i]], markersize=10)

plt.scatter(centers2D[:, 0], centers2D[:, 1], marker='x', s=200, linewidths=3, c='r')
plt.show()

Can someone see where I went wrong?


texts:
I'm new to deep learning, I am learning LSTM for my PhD work. This is a simple LSTM network for sequence classification.
This code is from MATLAB tutorial:

layers = [sequenceInputLayer(1)
    lstmLayer(5,'OutputMode','last')
    fullyConnectedLayer(3)
    softmaxLayer
    classificationLayer];

For simplicity, the dimension of the input sequence is 1, there are 3 classes.

I am trying to draw the diagram of this network. This is my attempt:

Is this right? Should the LSTM blue units be connected? The orange units are the softmax layer, is there any symbol the should be on each unit (like ∑)? The same question for each layer? Should be any extra layer to represent the "classificationLayer"? The fullyConnectedLayer is represented implicitly in the full connection in the last layer, Do I need to add any extra layer for this? Any other comments, please?


texts:
I'm looking for a good introductory book or course to big data analytics.
For the practical part, I'm particularly interested into using big data tools in R.

I would prefer a book, but other references are welcome.

Thanks!


texts: Average and standard deviations around timestamps

texts:
I have lots of events with timestamps like "15/07/2017  12:38:20" with accompanying information for each event. I want to find the length of time it takes for most events to occur on a given day, and over multiple days using averages and standard deviations.

I could calculate the time between first and last timestamp of the day, but there are outliers which skew the data, so by using averages and standard deviations I can implement the 68-95-99% statistical rule to calculate the time it takes for 68% of events to occur (or 95% or 99%), and add boxplots etc. The data is distributed normally.

I have extracted the time part of each timestamp (e.g. 12:38:20) and converted that into seconds (a day has 86,400) by multiplying the hour by 3,600 seconds, minutes by 60 seconds and then adding all the seconds together. 12:38:20 therefore is 45,500 seconds into the day.

So this works fine for a single day. For example:

Day 1: say 100 events occur.
Mean = 33,317 seconds into the day (so around 9.15am, i.e. the mean event occurred at this time)
Stdev.p = 3,378 seconds (1 stdev away from the mean, so around 56 minutes 18 seconds)
Stdev.p*2 = 6,757 seconds (between -1 and +1 stdev's away from the mean, this is the time it takes for 68% of events to occur, so around 1 hr 52 minutes)

All good so far. I can calculate this for every single day across my dataset.

My question is how to group multiple days together, so I can show the time it takes on average for a group of days (the whole week, month, quarter or whatever).

Do I take the average of the standard deviations - i.e. stdev for day 1 + stdev for day 2, divided by the number of days?
Or do I calculate the standard deviation for all events over days 1 and 2? Doing it this second way gives bigger durations across the two days.

As I'm typing this I'm feeling like its the first option. The second way mixes times from the first day with the second day, which is incorrect.

I think I've answered my own question, but would welcome any reassurance or comments to the contrary.


texts: regex to remove repeating words in a sentence

texts:
I am new to regex. I am working on a project where i need to replace repeating words with that word. for example:

  I need need to learn regex regex from scratch.

I need to change it to:

  I need to learn regex from scratch.

I can identify the repeating words using the following regex
\b(\w+)\b[\s\r\n]*(\l[\s\r\n])+

For substituting it, I need the word in the repeated word phrase.
pattern.sub(sentence, <what do i write here?>)


texts:
I have dataset where input is userId and day and Output is project activity and hour, each day some users are reporting for two projects, two activities and 4 hour each and some users only report for single project and activity.

I want output $\{
\{p1,a1,h1\},
\{p2,a2,h2\}\}$
for those users who reports for more than 1 project and activity a day,, i am not getting how to do and which algorithm to apply

Any help?


texts: How does real world machine learning production systems run?

texts:
Dear Machine Learning/AI Community,

I am just a budding and aspiring Machine Learner who has worked on open online data sets and some POC's built locally for my project. I have built some models and converted into pickle objects in order to avoid re-training.

And this question always puzzles me. How does a real production system work for ML algorithms?

Say, I have trained my ML algorithm with some millions of data and I want to move it to production system or host it on a server. In real world, do they convert into pickle objects? If so, it would be huge pickled file, isn't. The ones I trained locally and converted for 50000 rows data itself took 300 Mb space on disk for that pickled object. I don't think so this is right approach.

So how does it work in order to avoid my ML algorithm to re-train and start predicting on incoming data? And how do we actually make ML algorithm as a continuous online learner. For example, I built a image classifier, and start predicting the incoming images. But I want to again train algorithm by adding the incoming online images to my previously trained data sets. May be not for every data, but daily once I want to combine all received data for that day and re-train with newly 100 images which my previously trained classifier predicted with actual value. And this approach shouldn't effect my previously trained algorithm to stop predicting incoming data as this re-training may take time based on computational resources and data.

I have Googled and read many articles, but couldn't find or understand to my above question. And this is puzzling me every day. Do manual intervention is needed for production systems as well? or any automated approach is there for it?

Any leads or answers to above questions would be highly helpful and appreciated. Please let me know if my questions doesn't make sense or not understandable.

This is not a project centric I am looking for. Just a generic case of real world production ML systems example.

Thank you in advance!


texts: how to provide alternate for a word in corpus for word2vec model

texts:
I need to specify semantically close words in a corpus. For instance newspaper/magazine in the following:

I read [newspaper/magazine] yesterday.

In this example both words are close in meaning. I need to find these kind of words.

How can I find those words?


texts:
I'm trying to write a script to get the most similar rows in a certain dataframe, based on a single row. Using scikit-learn
The method I need is kneighbors, which kan return the indices of the closest lying rows in my dataframe if I call it on the NearestNeigbors algorithm.
The problem is that my data contains a lot of categorical variables, which NearestNeighbors can not handle out of the box. In previous projects I always got away with writing a pipeline that did preprocessing and onehotencoding to transform the data to integer/float values. But as not every estimator has a kneighbors method, a pipeline object does not implement a kneighbors method.
What is the best route to both:

Preprocess the categorical data so it gets onehotencoded and scaled
Query an unprocessed row to find the kneighbors of the queryrow.

So that unprocessed row should first get processed the same way the
training data was and after that I want to get the kneighbors which
show the "original" categorical values (so not the processed ones)

cars_data = ...

nbrs = NearestNeighbors(n_neighbors=50, algorithm='ball_tree').fit(pd.get_dummies(cars_data))

testpd = pd.DataFrame(columns=['make', 'model', 'body_type', 'transmission', 'fuel','km', 'kw', 'age_months', 'doors','seats', 'color','interior','airconditioning', 'lane_departure', 'led', 'gps', 'panoramic', 'openroof', 'parkassist','soundsystem','standheat','towhook','xenon', 'seatheat','alloywheels'])
testpd.loc[0] = ['bmw',       '116',        'berline',     'manual',   'benzine',   62450,  80, 10,5,5, 'black', 'fabric', 'YES', 'NO', 'NO', 'NO', 'NO', 'NO', 'YES', 'NO','NO', 'YES', 'NO', 'YES', 'NO'] #should be a variable that can be used to 'query' and get the 50 nearest neighbors

closest_neighbors = nbrs.kneighbors(testpd)

This obviously does not work as the testpd is not transformed. I want to do the same transformation as the train data on this testpd, but OneHotencoding and the scaling is not 'saved' and I'm unable to reapply that on this dataframe with a singular row.


texts: bias variance decomposition for classification problem

texts:
It is given that:

MSE = bias$^2$ + variance

I can see the mathematical relationship between MSE, bias, and variance. However, how do we understand the mathematical intuition of bias and variance for classification problems (we can't have MSE for classification tasks)?

I would like some help with the intuition and in understanding the mathematical basis for bias and variance for classification problems.

Any formula or derivation would be helpful.


texts: Masking seems not working for missing values problem in LSTM

texts:
I am trying to use LSTM to predict time series in keras. My input data shape is (1000,6,1)(samples,timesteps,features). There is some missing data in different timesteps. For example,[2,1,1]=NaN,[3,4,1]=NaN. First, I normalize all these data between (0,1)
Then I try to use fillNan to replace these Nan by special values(-1 for example). Finally, I use Masking to escape the special values.
My code is like that:

model = Sequential()
model.add(Masking(mask_value=1.5,input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(LSTM(32))
model.add(Dense(3))
model.compile(loss='mae', optimizer='adam')
model.fit(train_X, train_y, epochs=50, batch_size=72,validation_data=(test_X, test_y), verbose=2, shuffle=False)

When I choose different special values, the outcomes changed and I don't know where is the problem. train_X.shape=(700,6,1),train_y.shape=(300,3).
Is it because there is also NaN in the trian_y which is not Masked?


texts: Isn't one-hot encoding a waste of information?

texts:
I was just playing around with one-hot representations of features and thought of the following:

Say that we're having 4 categories for a given feature (e.g. fruit) {Apple, Orange, Pear, Melon}. In this case the one-hot encoding would yield:

Apple:  [1 0 0 0]
Orange: [0 1 0 0]
Pear:   [0 0 1 0]
Melon:  [0 0 0 1]

The above means that we quadruple the feature space as we go from having one feature to having four.

This looks like it's wasting a few bits, as we can represent 4 values with $\log_{2}4=2$ bits/features:

Apple:  [0 0]
Orange: [0 1]
Pear:   [1 0]
Melon:  [1 1]

Would there be a problem with this representation in any of the most common machine learning models?


texts: What does from_logits=True do in SparseCategoricalcrossEntropy loss function?

texts:
In the documentation it has been mentioned that y_pred needs to be in the range of [-inf to inf] when from_logits=True. I truly didn't understand what this means, since the probabilities need to be in the range of 0 to 1! Can someone please explain in simple words the effect of using from_logits=True?

model.compile(optimizer='adam',
              loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=['accuracy'])


texts: How can I efficiently represent statistics and visualizations for a large number of features?

texts:
Suppose I have a dataset with more than 50 features (could be 100 or 10,000, etc.). I'd like to create a summary table that has some statistics about each feature and plot the distributions. Normally, I'd use df.describe() and add various stats to that dataframe, but the resulting dataframe can be very large and take more effort to parse through and understand the features, plus it's tough to fit on a single-page in a Word document.
Plotting the features using a Seaborn Clustermap or Pairplot would be a typical way I'd approach it, but for the larger number of features, the plot gets too large to see relationships clearly. Using a for loop to plot histogram and bar plots would work but is messy and not efficient for datasets with more than 100 (my rough estimate and preference) features.
How can I efficiently represent these stats and plots for the large number of features?
If there is a better method in R, that'd be helpful too. I know the tableone package is really good (and there's a Python version), but again it gets hard to fit it in a single-page Word document. If my question is too subjective, I can try to rephrase it.


texts: ML approach to esimtate equilibrium price for a Real Estate Property

texts:
I am looking to calculate the equilibrium price, i.e an optimal price that I can set without affecting demand and maximize revenue.

I've gathered historical data: occupancy rates, asking rents for different property types by submarket / market and have a bunch of other data including local economic variables, new construction, location characteristics etc. Here's how I am thinking about estimating the demand function -

First predict demand, occupancy rates using Regression with fixed effects and historical prices for a particular property type in a given submarket. The training data for this is last 5+ years of historical data.

Approx. Model specification:

Dp = bo + b1*(Econ) + b2*Supply + b3*Loc + b4*Price t + b5*Price t-1 + b6*Price t-2

With this, I can get an estimate of how much price can change without significantly affecting quantity demanded. The assumption here is all else being equal. However, supply/vacancies changes.

So, my question is how would you model this? Are there any other analytical/ML approaches to model this and calculate equilibrium price?


texts: I'd like to change the log level of yarn. For example, when I run yarn node -list, I see a line that says INFO client.RMProxy: Connecting to Resource Manager.... I don't want to see that. Is there an easy way?


texts:
I have a huge data set with one of the columns named 'mail_id'. The mail_id is given in a very creepy format as shown below:
mail_id
DQ/4I+GIOz2ZoIiK0Lg0AkwnI35XotghgUK/MYc101I=
BL3z4RtiyfIDydaRYWX2ZXL6IX10QH1yG5ak1s/8Lls=
BL3z4RtiyfIDydaRYWX2ZXL6IX10QH1yG5ak1s/8Lls=
EHNBRbi6i9KO6cMHsuDPFjZVp2cY3RH+BiOKwPwzLQs=
K0y/NW59TJkYc5y0HUwDeAXrewYT0JQlkcozz0s2V5Q=
UGATDXARg7jMEInKH7oXgty2nwxnwD2l0OW/8Nsa0MI=
qE9zgWiITYA97RUiN4X/t9IVWLViLz+lKijaYegyBiQ=
BL3z4RtiyfIDydaRYWX2ZXL6IX10QH1yG5ak1s/8Lls=
4+EEK8RbNYwuFCHznY9XSRCV4Yek60bHVgnP3jtjjzk=

After doing a lot of analysis on my data, I have found that I cannot drop this feature set from my model so I have to convert it to something meaningful. Can anyone please explain me how to do this efficiently?


texts:
I am working on an Autoencoder in keras with the following setting: 185-86-32-2-32-86-185. The problem is that its accuracy is fluctuating. Also, it is giving a new accuracy at every run.
If this is due to the random uniform kernel initialization, how can I save the output values of the initializes in order to use them later?
My code:


texts: How Int8 (byte) operations can be useful for deep learning?

texts:
Nvidia is planing to add hardware support for int8 operations to their titan card and target deep learning. I am trying to understood how its useful and what types of network will benefit from this.

I know that FP16 instead of FP32 is what should be useful for DL, but not sure how int8 could do. There are some research that you can train with full FP32 precision and then round it to one byte - but this does not speedup or reduce memory footprint for training.


texts:
I am implementing a custom loss in keras, for example, a sum:

def custom_loss(y_true, y_pred):
    K.sum(y_true, y_pred)

Now, I want to normalize it by the batch size. Is it possible, to retrieve the batch size from y_true or y_pred?


texts:
I am practicing to use sklearn for decision tree, and I am using the play tennis data set:

play_ is the target column.

as per my pen and paper calculation of entropy and Information Gain, the root node should be outlook_ column because it has the highest entropy.

But somehow, my current decision tree has humidity as the root node, and look likes this:

My current code in python:

from sklearn.cross_validation import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score
from sklearn import tree
from sklearn.preprocessing import LabelEncoder

import pandas as pd
import numpy as np

df = pd.read_csv('playTennis.csv')

lb = LabelEncoder()
df['outlook_'] = lb.fit_transform(df['outlook'])
df['temp_'] = lb.fit_transform(df['temp'] )
df['humidity_'] = lb.fit_transform(df['humidity'] )
df['windy_'] = lb.fit_transform(df['windy'] )
df['play_'] = lb.fit_transform(df['play'] )
X = df.iloc[:,5:9]
Y = df.iloc[:,9]

X_train, X_test , y_train,y_test = train_test_split(X, Y, test_size = 0.3, random_state = 100)

clf_entropy = DecisionTreeClassifier(criterion='entropy')
clf_entropy.fit(X_train.astype(int),y_train.astype(int))
y_pred_en = clf_entropy.predict(X_test)

print("Accuracy is :{0}".format(accuracy_score(y_test.astype(int),y_pred_en) * 100))


texts: to extract multiple of n images using open cv in python

texts:
I am using opencv module in python to extract sequences of images(frames) from video (from a file)
My code for extracting images from video goes like this.

import cv2
import os
os.chdir('D:/Python images/videofolder')
cap = cv2.VideoCapture('D:/Python images/test_videos/testcutvideo1.avi')
cap.isOpened()
success,image = cap.read()
count = 1
success = True
while success:
     success,image = cap.read()
     cv2.imwrite("image%d.jpg" % count, image)     # save frame as JPEG file
count += 1

I want my code to select multiple of 'n' images
eg: to select & save multiple of 5 images(to extract multiple of 5) so that I have 5th, 10th, 15th... images.
Please guide me.


texts: Could perceptron use the standard logistic function as its activation function?

texts:
the perceptron is an algorithm for supervised learning of binary classifiers. A binary classifier is a function which can decide whether or not an input, represented by a vector of numbers, belongs to some specific class.

per this post, a perceptron could use some other functions as its activation function.

The standard logistic function is the logistic function with parameters (k = 1, x0 = 0, L = 1) which yields

${\displaystyle {\begin{aligned}f(x)&={\frac {1}{1+e^{-x}}}={\frac {e^{x}}{e^{x}+1}}={\tfrac {1}{2}}+{\tfrac {1}{2}}\tanh({\tfrac {x}{2}})\\\end{aligned}}}$

Could perceptron use the standard logistic function as the activation function? if yes, what is the difference between a perceptron with the standard logistic function and logistic function


texts: How to merge monthly, daily and weekly data?

texts:
Google Trends returns weekly data so I have to find a way to merge them with my daily/monthly data.

What I have done so far is to break each serie into daily data, for exemple:

from:

2013-03-03 - 2013-03-09 37

to:

2013-03-03 37
2013-03-04 37
2013-03-05 37
2013-03-06 37
2013-03-07 37
2013-03-08 37
2013-03-09 37

But this is adding a lot of complexity to my problem. I was trying to predict google searchs from the last 6 months values, or 6 values in monthly data. Daily data would imply a work on 180 past values. (I have 10 years of data so 120 points in monthly data / 500+ in weekly data/ 3500+ in daily data)

The other approach would be to "merge" daily data in weekly/monthly data. But some questions arise from this process. Some data can be averaged because their sum represent something. Rainfall for example, the amount of rain in a given week will be the sum of the amounts for each days composing the weeks.

In my case I am dealing with prices, financial rates and other things. For the prices it is common in my field to take volume exchanged into account, so the weekly data would be a weighted average. For financial rates it is a bit more complex a some formulas are involved to build weekly rates from daily rates.  For the other things i don't know the underlying properties. I think those properties are important to avoid meaningless indicators (an average of fiancial rates would be a non-sense for example).

So three questions:

For known and unknown properties, how should I proceed to go from daily to weekly/monthly data ?

I feel like breaking weekly/monthly data into daily data like i've done is somewhat wrong because I am introducing quantities that have no sense in real life. So almost the same question:

For known and unknown properties, how should I proceed to go from weekly/monthly to daily data ?

Last but not least: when given two time series with different time steps, what is better: Using the Lowest or the biggest time step ? I think this is a compromise between the number of data and the complexity of the model but I can't see any strong argument to choose between those options.

Edit: if you know a tool (in R Python even Excel) to do it easily it would be very appreciated.


texts:
Most problem with machine learning projects I have faced is the lack of data. The samples available are enough to disqualify rule based approach but not enough for a neural network to train.
For example, to train a neural network (or even fine tune a pretrained model) on a new entity in an NER system takes a thousands of different records. And the requirement of these thousands of records is to have enough variations to avoid overfitting.
Generally as human we can detect patterns by carefully observing the data, however it becomes humanly not possible to detect all patterns in inputs, and that is where deep learning comes into play of automatically detecting the patterns to make a hypothesis.
Now my question is, what are possible ways, using which limited data can be used to train a neural network with limited data. Let me add some inputs from my side, which I believe are not sufficient:

Data Augmentation : for images rotating, scaling and skewing. For textual data, repeating text with some masking and embedding/synonym replacement.
what else?


texts: How to prepare/optimise data for FP-Growth algorithm

texts:
I'm using spark mllib for FP-Growth algorithm for our ML model.
Description of my issue:
I have taken transactional data from our production database to mine the frequent brought items recommendation. Now when I run the data with this algo I always get recommendations on the products that are sold the most. For ex. For a t-shirt I get recommendations of t-shits itself instead of something like pants etc..
Now one thing which I have Noticed is the item t-shirt in above example is present in dataset for 65k times and pant is appearing only in 2k cases.
how do I now optimise the data to get a better prediction?


texts: Validation accuracy is fixed to 0.1000 for first epochs

texts:
I'm new to Deep Learning. A colleague gave me tfrecords for two image tf.data.Datasets, one for training and one for validation. I verified that the images are valid and that there is a 80/20 ratio between the two datasets. It is classification problem with 10 classes.
The problem I have is that the validation accuracy is fixed at exactly 0.1000 for the first 10-20 epochs while the validation loss varies in that time e.g. between 2.3 and 4.8. I have used a variety of out-of-the-box architectures such as ResNET50 or MobileNet (random initial weights) and lots of hyperparameter combinations (batch size, learning rate, number of epochs, optimizer) but the behavior is always the same. What could be the reason for this?


texts: How to understand image of Fourier transform?

texts:
When image is transformed using 2d fourier transform we can visualize the output as:

I think I understand correctly that those lines that come from the center represent high frequencies in horizontal/vertical direction when the size value of frequency related the to length of the line.

What the those lines that seem to connect the ends of those lines comming from the center mean?
How is value of sine/cosine visible on this image?


texts: What are some of the available methods for handling multi-label classification for longer sequences of text

texts: I am looking to solve a multi-class classification problem with long sequences of text with some rows having 1000's of tokens. Some of the state of the art methods such as BERT have a token limit and I was wondering what is currently being done to handle longer text sequences when dealing with classification?


texts: Computing Jaccard Similarity between two documents

texts:
Data Mining:

Compute the Jaccard similarity of D1 and D2 on 2-shingles is. Sim(D1,D2) =

D1 = the quick brown fox jumps over the lazy dog

D2 = jeﬀ typed the quick brown dog jumps over the lazy fox by mistake

Could you explain how to reach the conclusion, step-by-step? Thanks.


texts: Python Neural Network - running into x_test y_test fit errors

texts:
I have built a neural network and it worked fine with a small dataset of around 300,000 rows with 2 categorical variables and 1 independent variable, but was running into memory errors  when i increased it to 6.5 million rows. So I decided to modify the code and am getting closer but now I am running into an issue with fit errors.  I have 2 categorical variables and one column for the dependent variable of 1's and 0's(suspicious or not suspicious.  To start off the dataset looks like this:

DBF2
   ParentProcess                   ChildProcess               Suspicious
0  C:\Program Files (x86)\Wireless AutoSwitch\wrl...    ...            0
1  C:\Program Files (x86)\Wireless AutoSwitch\wrl...    ...            0
2  C:\Windows\System32\svchost.exe                      ...            1
3  C:\Program Files (x86)\Wireless AutoSwitch\wrl...    ...            0
4  C:\Program Files (x86)\Wireless AutoSwitch\wrl...    ...            0
5  C:\Program Files (x86)\Wireless AutoSwitch\wrl...    ...            0

My code followed/with the errors:

import pandas as pd
import numpy as np
import hashlib
import matplotlib.pyplot as plt
import timeit

X = DBF2.iloc[:, 0:2].values
y = DBF2.iloc[:, 2].values#.ravel()

from sklearn.preprocessing import LabelEncoder, OneHotEncoder
labelencoder_X_1 = LabelEncoder()
X[:, 0] = labelencoder_X_1.fit_transform(X[:, 0])
labelencoder_X_2 = LabelEncoder()
X[:, 1] = labelencoder_X_2.fit_transform(X[:, 1])

onehotencoder = OneHotEncoder(categorical_features = [0,1])
X = onehotencoder.fit_transform(X)

index_to_drop = [0, 2039]
to_keep = list(set(xrange(X.shape[1]))-set(index_to_drop))
X = X[:,to_keep]

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)

#ERROR
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/sklearn/base.py", line 517, in fit_transform
    return self.fit(X, **fit_params).transform(X)
  File "/usr/local/lib/python2.7/dist-packages/sklearn/preprocessing/data.py", line 590, in fit
    return self.partial_fit(X, y)
  File "/usr/local/lib/python2.7/dist-packages/sklearn/preprocessing/data.py", line 621, in partial_fit
    "Cannot center sparse matrices: pass `with_mean=False` "
ValueError: Cannot center sparse matrices: pass `with_mean=False` instead. See docstring for motivation and alternatives.

X_test = sc.transform(X_test)

#ERROR
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/sklearn/preprocessing/data.py", line 677, in transform
    check_is_fitted(self, 'scale_')
  File "/usr/local/lib/python2.7/dist-packages/sklearn/utils/validation.py", line 768, in check_is_fitted
    raise NotFittedError(msg % {'name': type(estimator).__name__})
sklearn.exceptions.NotFittedError: This StandardScaler instance is not fitted yet. Call 'fit' with appropriate arguments before using this method.

If this helps i printed the X_train and y_train:

X_train
<5621203x7043 sparse matrix of type '<type 'numpy.float64'>'
with 11242334 stored elements in Compressed Sparse Row format>

y_train
array([0, 0, 0, ..., 0, 0, 0])


texts: How Do I Perform Gradient Descent for Discrete Predictions

texts:
I have a dataframe, df, that is of the format:

Dimension, 1, 2, ..., 55

Dim1, 0.22, 0.152, ..., 0.10

...

Dim20, 0.52, 0.89, ..., 0.09

And I have a training set:

participant_1 :{Dimension: Dim12, response: {1,5,24,6,12}}
participant_2 :{Dimension: Dim5, response: {1,5,24,30,32}}

I would like to find weights w_1, w_2, ..., w_55 such that I can weight the columns of df corresponding to a participants response est, add the columns together and the row with the highest resulting value should be the participants dimension value.

What is the best method to find the set of weights in Python?

I am also looking for advice on how to format questions here.

Thank you!!


texts: MSE errors on autoencoder for dim reduction decreases in a weird patteren and I would love some help to dechyper it

texts:
I'm training a denoising autoencoder right now to reduce the dimension of a feature vector I designed of dim 58 to a latent space of dim 10, or less hopefully. I'm having a hard time understanding what does the following error decreasing pattern imply :
]1
My intuition tells me the gradient descent is "crawling" toward a valley for a few epochs, once he reached that valley he quickly improve and then crawl again. So if I'm correct, does it mean my learning rate is too low? If I'm wrong I would love some help.

My final goal is to use Diffusion maps algorithms on a low dim dataset to graph it and find meaningful clusters and patterns.
I'm also adding 0.05 Gaussian noise to each feature vector during the start of each forward pass.
While each feature range is a bit different, all of them range between [5,-2.5]
LATENT_SPACE_DIM=10
The feature vector is extracted from 50k users rating in the Netflix dataset

Code in pytorch for the AE architture :
Class Denoising_AE(nn.Module):
   def __init__(self):

    super(Denoising_AE, self).__init__()

    # encoder
    self.enc0 = nn.Linear(in_features=58, out_features=45)
    self.enc1 = nn.Linear(in_features=45, out_features=30)
    self.enc2 = nn.Linear(in_features=30, out_features=15)
    self.enc21 = nn.Linear(in_features=15, out_features=LATENT_SPACE_DIM)

    # self.enc6 = nn.Linear(in_features=10, out_features=8)
    # self.enc = nn.Linear(in_features=8, out_features=6)
    #
    #
    # # decoder
    # self.dec = nn.Linear(in_features=6, out_features=8)
    self.dec0 = nn.Linear(in_features=LATENT_SPACE_DIM, out_features=15)
    self.dec1 = nn.Linear(in_features=15, out_features=30)
    self.dec2 = nn.Linear(in_features=30, out_features=45)
    self.dec3 = nn.Linear(in_features=45, out_features=58)


texts: Correcting Datasets with artificially low starting values

texts:
I am working on a project where we would like to take the ratio of two measurements A/B and subject these ratios to a ranking algorithm. The ratio is normalized prior to ranking (though the ranking/normalization are not that import to my question).

In most cases measurement A (the starting measurement) is a count with values greater than 1000.  We expect an increase for measurement B for positive effects and a decrease in measurement B for negative effects.

Here is the issue, some of our starting counts are nearly zero which we believe is an artifact of experimental preparation.  This of course leads to some really high ratios/scaling issues for these data points.

What is the best way to adjust these values in order to better understand the real role in our experiment?

One suggestion we received was to add 1000 to all counts (from measurement A and B) to scale the values and remove the bias of such a low starting count,  is this a viable option?  Thank you in advance for your assistance, let me know if I am not being clear enough.


texts: Can not install spacy package on windows 10 via pip

texts:
I have below environment.
OS: Windows 10
Python: Python 3.7.4
PIP: pip 19.3.1
I am trying to install spacy in my windows 10 OS. It gives me below error.
 ERROR: Command errored out with exit status 1:
   command: 'd:\rajesh\python\env1\scripts\python.exe' 'd:\rajesh\python\env1\lib\site-packages\pip' install --ignore-installed --no-user --prefix 'C:\Users\rajesh.das\AppData\Local\Temp\pip-build-env-vna552d_\normal' --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'thinc<7.4.0,>=7.3.0' 'cymem<2.1.0,>=2.0.2' 'preshed<3.1.0,>=3.0.2' wheel 'cython>=0.25' 'murmurhash<1.1.0,>=0.28.0'
       cwd: None
  Complete output (460 lines):
  Collecting thinc<7.4.0,>=7.3.0
    Using cached https://files.pythonhosted.org/packages/d4/38/f79bb496ced36f8d69cdbdfe57a322205582ed9508bda5bd0227969d5a77/thinc-7.3.1.tar.gz
  Collecting cymem<2.1.0,>=2.0.2
    Using cached https://files.pythonhosted.org/packages/ce/8d/d095bbb109a004351c85c83bc853782fc27692693b305dd7b170c36a1262/cymem-2.0.3.tar.gz
  Collecting preshed<3.1.0,>=3.0.2
    Using cached https://files.pythonhosted.org/packages/5f/14/de231123ddbe0bf12bd9b1993122d67f22859643bee4dad3b6ce91986336/preshed-3.0.2.tar.gz
  Collecting wheel
    Using cached https://files.pythonhosted.org/packages/00/83/b4a77d044e78ad1a45610eb88f745be2fd2c6d658f9798a15e384b7d57c9/wheel-0.33.6-py2.py3-none-any.whl
  Collecting cython>=0.25
    Using cached https://files.pythonhosted.org/packages/6d/da/b48f8a1778a82d24d1c3a543d9db8461374f0c70c0145f0b5a8664dbc122/Cython-0.29.14-cp37-cp37m-win32.whl
  Collecting murmurhash<1.1.0,>=0.28.0
    Using cached https://files.pythonhosted.org/packages/22/e9/411be1845f1ac07ae3bc40a4b19ba401819baed4fa63b4f5ef28b2300eb4/murmurhash-1.0.2.tar.gz
  Collecting blis<0.5.0,>=0.4.0
    Using cached https://files.pythonhosted.org/packages/98/5a/f9b8a78e3d1fdde1b0215413d88ab55d907ab81f95b62418a6e9cda30dec/blis-0.4.1.tar.gz
  Collecting wasabi<1.1.0,>=0.0.9
    Using cached https://files.pythonhosted.org/packages/ff/ef/e8266e158ed32bf5f723fac862b6518833d0b53ca183165a8718f212c0d5/wasabi-0.4.2-py3-none-any.whl
  Collecting srsly<1.1.0,>=0.0.6
    Using cached https://files.pythonhosted.org/packages/1e/5b/f83e478fbf44c0d04f182e56db0871a53f5096febb8eb0be7cec75dca73b/srsly-0.2.0.tar.gz
  Collecting numpy>=1.7.0
    Using cached https://files.pythonhosted.org/packages/ce/ad/2e88f36b56f64f70c081b32fa5512dacedf12005ccb0c2d300d44dcc1215/numpy-1.17.4-cp37-cp37m-win32.whl
  Collecting plac<1.2.0,>=0.9.6
    Using cached https://files.pythonhosted.org/packages/86/85/40b8f66c2dd8f4fd9f09d59b22720cffecf1331e788b8a0cab5bafb353d1/plac-1.1.3-py2.py3-none-any.whl
  Collecting tqdm<5.0.0,>=4.10.0
    Using cached https://files.pythonhosted.org/packages/7f/32/5144caf0478b1f26bd9d97f510a47336cf4ac0f96c6bc3b5af20d4173920/tqdm-4.40.2-py2.py3-none-any.whl
  Building wheels for collected packages: thinc, cymem, preshed, murmurhash, blis, srsly
    Building wheel for thinc (setup.py): started
    Building wheel for thinc (setup.py): finished with status 'error'
    ERROR: Command errored out with exit status 1:

Also I was trying to install dependency package of spacy thinc, It also gives same error.
Can anyone give some idea?


texts: Is label-encoded data quantitative or qualitative?

texts: If you label-encode something which is qualitative, like brand of toothpaste or colour of hair, would you describe the resulting data as quantitative since it is now expressed in numbers? Or would you continue to describe the data as qualitative, since it is still describing the same underlying information?


texts:
I have got a GTX 1650TI GPU and am trying to run Tensorflow using the GPU, however Tensorflow does not detect the GPU. I am using Tensorflow version 2.3.0 with cuda 10.1 and cudnn 7.6. Trying to detect the GPU gives the following results:
import tensorflow as tf
print("hardware", len(tf.config.experimental.list_physical_devices('GPU')))
hardware 0

or
import keras
from tf.python.client import device_lib
print(device_lib.list_local_devices())

[name: "/device:CPU:0"
device_type: "CPU"
memory_limit: 268435456
locality {
}
incarnation: 8086797182931046935
]


texts: Experience Replay, must return minibatch back to Memory Bank?

texts:
During Experience Replay, we are randomly gathering a minibatch from the Memory bank. We then use the minibatch to correct our NeuralNetwork q-value function approximator.

When done, should we return the minibatch back to the memory bank, or should we throw it out?

Doing the former seems to grow the memory indefinitely, and might result in some state being sampled a few extra times in the future, if the state is lucky and ends up in future mini batches.


texts: Can a convention convolution neural network train correctly with different training image size and ratio?

texts:
For example the task  of transformation, the model consist convolutional layer and pooling layer only, take input of image, and output a feature map (loss MSE, trying to produce feature map that exactly match feature map in label)

The forward pass can take any image size and output a feature with size = [w/(n_max_pooling*2), h/(n_max_pooling*2)]

Thus, with batch size of 1, there are no "dimension mismatch error" when we feed different image size, different ratio. The network just output a feature map, due to the nature of convolutional layer and max pooling layer.

For example: (batch_size, channel, width, height)

Sample 1:  1x3x128x128 => target 64x64
Sample 2: 1x3x768x1024 =>
target 368x512

....

I would like to know in that case of inconsist input shape, will the network be able to train and update it weight correctly ?


texts: Why not use the anchor boxes for the grid based search to detect objects directly in YOLO algorithm?

texts:
The YOLO algorithm does a grid search based on the grid squares (I'm not sure if they ALWAYS are squares) and when an object is detected in one of the squares, check for the maximum 'Intersection over Union' with one of the anchor boxes and then draw the boundary around the object.

My question is, is computational inefficiency the sole reason why we don't simply run these anchor boxes instead of first running the squares around and then checking with the anchor boxes? I am just starting out with this, so I haven't implemented anything of yet. I don't know how costly that inefficiency will be.

Also, why the name 'You Only Look Once'? Is it just the researchers having a fun time or does it have a reason?


texts:
In GoogLeNet (this link), there is 'v' notation in Figure3 like '1X1+1(v)'.
I don't know the meaning of 'v'.

Also, I understood 's' as stride. But, I don't know the reason why plus operation is used in '1x1+1(s)'.


texts: PCA vs.KernelPCA: which one to use for high dimensional data?

texts:
I have a dataset which contains a lot of features (>>3). For computational reasons, I would like to apply a dimensionality reduction. At this point I could use different techniques:

standard PCA
Kernel PCA
LLE
...
My problem is to choose the right approach since the number of features is so high that I cannot know beforehand what the distribution of points is like. I could do it only if I have 3D data, but in my case I have much more than that.

I know for example that if the set of points was lineary seperable I could use standard PCA; if it was somehow a sort of concentric circles like shape, then KernelPCA would be a better option.

Therefore how can I know beforehand which dimensionality reduction technique I need to use for high dimensional data?


texts: How to get the probability of belonging to clusters for k-means?

texts:
I need to get the probability for each point in my data set. The idea is to compute distance matrix (first column contsins distances to first cluster, second column conteins distances to second cluster and etc). The closest point has probability = 1, the most distant has probability = 0.
The problem is linear function (like MinMaxScaller) have output where almost all points have almost the same probability.

How to choose nonlinearity for this task? How to automatizate this process on python? For example for the most closest point p=1, for the most distant point that belongs to cluster p=0.5, for the most distant point p is almols 0.

Or you can propose another methods for computing this probability.


texts: How to use text as an input for a neural network - regression problem? How many likes/claps an article will get

texts:
I am trying to predict the number of likes an article or a post will get using a NN.
I have a dataframe with ~70,000 rows and 2 columns: "text" (predictor - strings of text) and "likes" (target - continuous int variable). I've been reading on the approaches that are taken in NLP problems, but I feel somewhat lost as to what the input for the NN should look like.
Here is what I did so far:

Text cleaning: removing html tags, stop words, punctuation, etc...
Lower-casing the text column
Tokenization
Lemmatization
Stemming

I've assigned the results to a new column , so now I have "clean_text" column with all the above applied to it. However, I'm not sure how to proceed.
In most NLP problems, I have noticed that people use word embeddings, but from what I have understood, it's a method used when attempting to predict the next word in a text. Learning word embeddings creates vectors for words that are similar to each other syntax-wise, and I fail to see how that can be used to derive the weight/impact of each word on the target variable in my case.
In addition, when I tried to generate a word embedding model using the Gensim library, it resulted in more than 50k words, which I think will make it too difficult or even impossible to onehot encode. Even then, I will have to one hot encode each row and then create a padding for all the rows to be of similar length to feed the NN model, but the length of each row in the new column I created "clean_text" varies significantly, so it will result in very big onehot encoded matrices that are kind of redundant.
Am I approaching this completely wrong? and what should I do?


texts: Why does my Multilayer Perceptron only classify linearly?

texts:
I wrote a multilayer perceptron in Python. I'm trying to get it to do nonlinear classification. It has two hidden layers, so it should be perfectly capable. Unfortunately, it only seems to be able to classify linearly. I am hesitant to call this "underfitting", because I am yet to properly achieve non-linear classification, even a non-linear classification that is an underfit. In the below graph, I attempted to classify points above and below the sine function, but it just drew a straight line through the dataset.

Points of uncertainty (outputs around 0.5) are in yellow.

I am using the sigmoid as the activation function, which is simply 1/(1+((math.e) ** (-sum))).

I use two input nodes, twenty hidden nodes in two hidden layers (ten each) and one output node. I have a bias on each hidden layer and one on the output node. My input is not normalized.

I feedforward hundreds of random points and calculate the difference between the output and desired. I calculate the mean-squared error and preserve the lowest mean-square error in each generation of my evolutionary algorithm. I won't put that on display here, since it seems to work pretty well in getting the error low, although my MSE function might be incorrect.

d=0
error = 0
while d < (tPoints): # while all the points in the list have not been feedforwarded
    output1 = feedForward(n, [xValsAbove[d], yValsAbove[d]]) # feedforwards a point above the function, returns output (0,1)
    output2 = feedForward(n, [xValsBelow[d], yValsBelow[d]]) # feedforwards a point below the function, returns output (0,1)

    if output1 < 0.99:
        error += (output1)**2
    if output2 > 0.01:
        error += (1-output2)**2
    d+=1
error = error/tPoints
return error # return mean-squared error

My neurons do linear regression lines just fine. Since I have an MLP with a few layers, I wrote an algorithm for feedforwarding. It just takes the outputs of the previous layer as inputs to the next layer.

def feedForward(n, inputs): # (inputs, inputList, hidden1, hidden2, outputs, bias):

    inputList = n.getNeuronList()[0]
    hidden1 = n.getNeuronList()[1]
    hidden2 = n.getNeuronList()[2]
    outputs = n.getNeuronList()[3]
    bias = n.getNeuronList()[4]

    FFInputs = []
    FFHidden1 = []
    FFHidden2 = []
    FFOutputs = []

    for inp in inputList:
        FFInputs.append(inp.feedForward(inp.getWeights(), inp.getInputs()))
    for hidden in hidden1:
        FFHidden1.append(hidden.feedForward(hidden.getWeights(), FFInputs, bias[0]))
    for hidden in hidden2:
        FFHidden2.append(hidden.feedForward(hidden.getWeights(), FFHidden1, bias[1]))
    for out in outputs:
        FFOutputs.append(out.feedForward(out.getWeights(), FFHidden2, bias[2]))

    return FFOutputs

Graph done with matplotlib.

Does anyone experienced in ANNs have any idea why it would do this? Should I only use one layer? I am evolving the weights with an evolutionary algorithm, so should I just do more generations? Should I use a different activation function?


texts: AdaBoost - Ensemble model perform poor than best weak classifier

texts: Can Adaboost's ensemble classifier perform worse than the best of the weak learners considered? If so when in what case of weak learner the ensemble learning does not perform better?


texts: How to specify version for dependencies so that each one is compatible and stays within a size limit?

texts:
I am trying to deploy a web app to Heroku. The free tier is limited to 500 MB.
I am using my resnet34 model as a .pkl file.
I create model with it using the fastai library.
This project requires torch and torchvision as dependencies.
But not specifying the dependency will download the latest version of torch which alone is 750 MB and exceeds the memory limit.
So, I specify torchvision version as 0.2.2 and specify the wheel for torch for v1.1.0 in the requirements.txt file. Now, this gives rise to other problems.
For example, I got this error message from Heroku build log-

ERROR: torchvision 0.2.2 has requirement tqdm==4.19.9, but you'll have tqdm 4.48.0 which is incompatible.

But if I specify tqdm's version to 4.19.9, some library called spacey becomes incompatible and the app fails. If I install more recent version of torchvision, the app size will exceed permitted limits.
How do I get around this problem? How do I find out which dependency versions are compatible with each other, and the total size doesn't exceed 500 MB? Is there an easy way to do that?


texts: Coloring clusters so that nearby clusters have different colors

texts:
I have clustered a large number of points (~3000) into (~400) clusters.  I want to plot the data and visualize the clusters.  I want to make sure that nearby clusters have different colors.  Can anyone recommend an approach to coloring the clusters?
This is a conceptual question, but I'm most interested in solutions in python or R.