texts: Feature extraction in time series data as input variables for machine learning algorithms

texts:
I have worked with time series data to predict the defect in production lines. I want to extract the feature from time series data as inputs variables for machine learning algorithms (such as support vector machine). My dataset looks like:

Timestamp  Pressure
t0           x0
t1           x1
t2           x2
.             .
.             .
.             .
tn           xn

There is a threshold b. If xt>b that means the defect happened. My goals are to extract features from time series dataset above and put it into algorithms to predict the value of time stamp tn+1, tn+2 (short-term) and long-term tn+10. However, I have not yet found the way to extract feature and how to bring the threshold into the algorithm. Could anyone suggest me how to deal with it? Thank you.


texts: Why do we need the sigmoid function in logistic regression?

texts:
What is the purpose of the logistic sigmoid function as it is used in logistic regression? Why does it need to be part of the hypothesis function h(x) ?

As I understand it, the logistic sigmoid function gives the probability that a certain input vector x is contained within a class C1 for a label y. In the binary-class case, it seems that if h(x) >= 0,5, we say that x belongs to one class, otherwise it belongs to the other. In the logistic regression model, our hypothesis function h(x) is of the form g(p^T * x), where p is the parameter vector (p^T is the transpose) and g is the sigmoid function.

Since the y-intercept of the logistic sigmoid is 0.5, saying that h(x) >= 0.5 is the same as saying p^T * x >= 0.

What I'm getting at is why do we need the logistic sigmoid function at all to define some threshold for separating the classes? Why not just let the hypothesis function be of the form h(x) = p^T * x, and claim that y = 1 if p^T * x >= 0? Why complicate things unnecessarily with the logistic sigmoid?


texts: Is multi-class classification the same with categorical classification?

texts:
I am working on a project with image classification with 5 different classes and I was wondering if categorical classification can be used as a synonym for multi-class classification.

PS. Forgive me for the naive question


texts: Proper Understanding of Condensed Nearest Neighbor

texts:
I have a question regarding the Condensed Nearest Neighbors algorithm:

Why am I returning Z, which if I understand correctly, is the array of all of the misclassified points? Wouldn't I want to return the points that were classified correctly? What benefit does this give me in returning all the points I got wrong?


texts:
I have plots of feature vs target values as

and

And many more. I have used Tree based models and able to get high accuracy but not able to improve further. Any suggestion ? almost every feature is like this.


texts: Preprocessing of Sudoku Dataset from Kaggle

texts:
Dataset: https://www.kaggle.com/bryanpark/sudoku

I would like to create a neural network for this Dataset.

Feature:

X:  [ '004300209005009001070060043006002087190007400050083000600000105003508690042910300'
 '040100050107003960520008000000000017000906800803050620090060543600080700250097100']

Output:

Y:  [ '864371259325849761971265843436192587198657432257483916689734125713528694542916378'
 '346179258187523964529648371965832417472916835813754629798261543631485792254397186']

The zeros represent a blank box, and the data is a flattened grid of 9x9.

I tried using the following code, but I found out that the data needs a significant amount of preprocessing.

def preprocess():

data = pd.read_csv('Sudoku/sudoku.csv')
print('Data: ', data.head())

x = data[data.columns[0]].values
y = data[data.columns[1]].values

x = x.reshape(-1, 1)
y = y.reshape(-1, 1)

print('\nx: ', x[0])
print('\ny: ', y[0])

return x, y

x, y = preprocess()
train_x, test_x, train_y, test_y = train_test_split(x, y, test_size=0.2)

scl = StandardScaler()
train_x = scl.fit_transform(train_x)
train_y = scl.fit_transform(train_y)
test_x = scl.fit_transform(test_x)
test_y = scl.fit_transform(test_y)

model = Sequential()
model.add(Dense(100, input_dim=1, activation='relu'))
model.add(Dense(50, activation='relu'))
model.add(Dense(1, activation='linear'))
model.compile(optimizer='adam', loss='mean_squared_error')
model.fit(train_x, train_y, epochs=3, validation_split=0.1, verbose=2)

I would like to know how to work with this data and of course, also the structure of my neural network.


texts: Bandwidth selection Kernel Density Estimation

texts: I want to do KDE on data that are not necessarily normal using Gaussian kernels. In KDE in wikipedia an expression for the bandwidth is given when the underlying distribution of the data is gaussian. I am looking for a similar bandwidth selection for non-normal data.


texts: Deep Learning models with top-down transfer

texts:
Are there Deep Learning models that utilize some sort of top-down information transfer (not only during training)?

For example, if the model is used for speech recognition. The lower layers would listen to the samples and say: "The word 'one', the word 'two', I can't understand what this sample is". And then an upper layer would suggest: "Could it be the word 'three', that is not uncommon after 'one' and 'two'". And then the lower layers would be like: "Now that you said it! Of course it's 'three'"

I'm pretty sure that an architecture like this is a requirement for reliable speech recognition, especially in noisy environments.


texts: Is there an algorithm for sampling shortest paths?

texts: I have a triangle matrix NxN of distances between vertices (vertex i connected only with vectices j>i) and I'd like to sample path from first to the last and use it as a training sample. Is there an algorithm to do it? Would swapping min->sample in A* be sufficient?


texts: AUC ROC Curve multi class Classification

texts:
Here is the part of the code for ROC AUC Curve calculation for multiple classes.
n_classes= 5

y_test = [0,1,1,2,3,4]   #actual value
pred1 = [0,1,1,1,3,4]    #predicted value

fpr = dict()
tpr = dict()
roc_auc = dict()

for i in range(n_classes):
    fpr[i], tpr[i], _ = roc_curve(np.array(pd.get_dummies(y_test))[:, i], np.array(pd.get_dummies(pred1))[:, i])
    roc_auc[i] = auc(fpr[i], tpr[i])

Error :

I understand that there are only 4 classes are predicted 0,1,3,4 in pred1 and 5 classes 0,1,2,3,4 in the y_test. So this error arises.
How to solve this error.


texts: Model should predict the same value every time for the same input

texts: I have used a random forest model for prediction of prices. Should the model be predictable in its behavior? By this, I mean that I'm not changing the model and the input, Will the predicted value always be the same?


texts: Neural net classifier outputting extreme probabilities

texts:
I am training a multi-label neural network text classifier (i.e. a given sample can have more than one label; most samples have exactly 1 label though):

Single-layer BiLSTM producing a sequence of (16, 512) vectors with dropout of 0.2
Self attention over each output vector of LSTM to produce a single 512-dimension "sentence vector"
L2 normalize sentence vector
ReLU layer on the normalized vector with about 750 units
Dropout of 0.3
Final sigmoid output layer to output a probability for each of the K classes (K~1000)

Firstly, does L2 normalization help or make things worse?

Secondly, this model is quite accurate but outputs very extreme "probabilities" for each class for the samples - very often < 0.00001 or > 0.9999 probability, instead of something less extreme (like say 0.8 or 0.2 or whatever). Any idea of what might be going wrong? If this is overfitting -- I'm using quite a bit of dropout too but that didn't help either. How can I remedy this?


texts: Statistical significance test in deep learning for regression problems

texts:
I was reading the tutorial "Statistical Significance Test for comparing ML algorithms", where it suggests to use k-fold and apply the appropriate statistical test.

Suppose that I have a train set and a test set and two deep learning neural networks in a regression problem. Since the training in deep learning takes lots of time, the test procedure using k-fold would be very costly regarding computation resources.

I was wondering if it is possible to apply the statistical tests only in the results of the test set.

For example, if the test set has 1000 samples, the two neural networks would be trained in the same training set and then apply Wilcoxon in the results of the two networks over the 1000 points of the test set, is that correct? Or do I always need to perform the k-fold? Besides can I use the MSE in each test point and compare those results?


texts: Spliting keras model into multiple GPU's

texts:
Dear fellow Data Scientists.
I'm having a problem with splitting model into multiple GPU's. I have read something about "towering" in native tensorflow but my whole architecture is already written in keras (tensorflow backend of course). Keras as far as I know only supports data paralellism which is useless while operating on images bigger than 1760x1760 in my case(Yolo architecture). I'm asking for advice, how could I achieve this without using native tensorflow I must run this model on 4500x4500 images and I can use up to 4 Tesla k40 (11GB) GPU's.

EDIT:
I'm already using batch = 1


texts:
Im doing a 2-class classification project for an imbalanced data set.
The imbalance is about 18%/82%. Im noticing a huge improvement in F1-score when I under-sample; from 16% without under-sampling to 49% with under-sampling. Im wondering if this is possible or that I am doing something wrong? Before under-sampling I split the data set in a train- and test set, apply outlier detection and normalisation (normalisation first on train set, then on test using the scales obtained from train set). Does anybody have any idea, if this improvement is too good to be true, what could be going wrong?? Many thanks in advance!


texts: Can you apply technical analysis indicators to general data?

texts: I collect data, for example, "how many people mentioned 'Bitcoin' on an investing community", I take the data for daily, so it's like "5/1,231;5/2,156;5/3,223" and so on. So basically not relevant to financial market price actions, but just a type of general data. Here as a newbie to the world of data science, I wonder the indicators which called "technical indicators" can be applied to such general data. For example, I can see sometimes people use "(simple) moving average" to general data, but then "moving average crossover" (e.g. 50sma crossing up 100sma) is effective to them? Or how about RSI(Relative Strength Index)? I can know the uptrend data could go down if I see divergence on the data and its RSI? Other major technical indicators?


texts: Problem with chain rule in softmax layer when differentiated separately

texts:
I have some problems with backpropagation in softmax output layer. I know how it should work but if I try to apply the chain rule in the classical way, I get different results compared to when Softmax is derivated with Cross-Entropy error. Here's an example from a network with a single data-point with 2 output neurons using Softmax as the activation function.
$$ Z^{O} := (1.75, 1.75) $$
$$ A^{O} := (0.5, 0.5) $$
Here $ Z^{O} $ is the layer output before applying the SoftMax activation function, while $ A^{O} $ is the activation - guess - of the network.

If I were to derive the SoftMax activation function with the CE error function would give me:
$$ \frac{\partial{E(A^{O},T)}}{\partial{Z^{O}}} = \frac{\partial{\sum_{i=0}^{n}{-T_{i}*ln(A^{O})}}}{\partial{Z^{O}}} = \frac{\partial{\sum_{i=0}^{n}{-T_{i}*ln(Softmax(Z^{O}))}}}{\partial{Z^{O}}} = A^{O} - T $$
With:
$$ A^{O} = \begin{pmatrix}0.5&0.5\end{pmatrix} $$
$$ T = \begin{pmatrix}0&1\end{pmatrix} $$
This yields:
$$ A^{O} - T = \begin{pmatrix}0.5&-0.5\end{pmatrix} $$

This is fine. Now I need to get the same result when applying the chain rule and differentiating the error function and the activation function separately:
$$ \frac{\partial{E(A^{O},T)}}{\partial{A^{O}}} * \frac{\partial{A^{O}}}{\partial{Z^{O}}} $$

The first part is trivial:
$$ \frac{\partial{E(A^{O},T)}}{\partial{A^{O}}} = \frac{\partial{\sum_{i=0}^{n}{-T_{i}*ln(A_{i}^{O})}}}{\partial{A^{O}}} = -\frac{T_{i}}{A_{i}^{O}} = \begin{pmatrix}-\frac{0}{0.5}&-\frac{1}{0.5}\end{pmatrix} = \begin{pmatrix}0&-2\end{pmatrix} $$

The second part is of course a matrix of $ Softmax^{'}(Z^{O}) $:
$$ \frac{\partial{Softmax(Z^{O})}}{\partial{Z^{O}}} = \begin{pmatrix}Z^{O}_{0}(1-Z^{O}_{0})&-Z^{O}_{0}Z^{O}_{1}&...&-Z^{O}_{0}Z^{O}_{n}\\-Z^{O}_{1}Z^{O}_{0}&Z^{O}_{1}(1-Z^{O}_{1})&...&-Z^{O}_{1}Z^{O}_{n}\\...&...&&...\\-Z^{O}_{n}Z^{O}_{0}&-Z^{O}_{n}Z^{O}_{1}&...&Z^{O}_{n}(1-Z^{O}_{n})\end{pmatrix} $$

def df2(d):
    def softmax_prime(f):
        jacobian = np.diag(f)
        for i in range(len(jacobian)):
            for j in range(len(jacobian)):
                if i == j:
                    jacobian[i][j] = f[i] * (1 - f[i])
                else:
                    jacobian[i][j] = -f[i]*f[j]
        return jacobian
return np.apply_along_axis(softmax_prime, axis=1, arr=d)

Which for the above $ \begin{pmatrix}1.75&1.75\end{pmatrix} $ yields:
$$ \frac{\partial{A^{O}}}{\partial{Z^O}} = SoftMax'(\begin{matrix}1.75&1.75\end{matrix}) = \begin{pmatrix}-1.3125&-3.0625\\-3.0625&-1.3125\end{pmatrix} $$

Now, according to the chain rule, if I multiply these two together I would get $ \delta^{O} $ for the output layer and it should be the same as the above $ A^{O} - T $. However this gives:
$$ \frac{\partial{E(A^{O},T)}}{\partial{A^{O}}} * \frac{\partial{A^{O}}}{\partial{Z^{O}}} = \begin{pmatrix}0&-2\end{pmatrix} * \begin{pmatrix}-1.3125&-3.0625\\-3.0625&-1.3125\end{pmatrix} = (6.125, 2.625) $$

Which is different from the the above original calculation. I'm sure I made a mistake, but I can't see where. The deltas must match that I'm sure about, and fair enough if I use the chain rule like this the error will increase for each epoch.

Can anyone tell me where I went off?


texts: (Feature selection) In which cases it is legitimate to remove features manually?

texts:
I am dealing with the feature in which only one category takes up about 90%, the instances of more than 30 other categories are sparse.

Is it reasonable to remove this feature before building an estimator?
I've experimented with the case when including and excluding this feature, and the later case showed a slightly(very slightly...) better performance.
I am wondering if there is any machine learning principle, such as we should avoid 'human learning' and just let 'machine learning' do. What would be the best way to deal with this sort of features? I am currently using Scikit-learn.

I am a total beginner in data science, so any opinion will be valuable to me. :)


texts: Split a string and create a news columns with parents

texts: let's say I have this dataframe, and I would like to split it into several after (.) and have it like that, with levels


texts: Why is there a $2$ at the denominator of the mean squared error function?

texts:
In the famous Deep Learning Book, in chapter 1, equation 6, the Quadratic Cost (or Mean Squared Error) in a neural network is defined as

$ C(w, b) = \frac{1}{2n}\sum_{x}||y(x)-a||^2 $

where $w$ is the set of all weights and $b$ the set of all biases, $n$ is the number of training inputs, x is the set of all training inputs, y(x) is the expected output of the network for input x, and $a$ is the actual output of the network for input $x$, with respect to $w$ and $b$.

Most of this formula seems very clear to be, except the $2$ in the denominator. If I understand it correctly, we are summing up the squared vector length of (the actual output minus its expected output), for each training input (giving us the total squared error for the training set) and then divide this by the number of training samples, to get the mean squared error of all training samples. Why do we divide this by $2$ as well then?

In other places I've seen that Andrew Ng's lecture defines the Mean Square cost in a similar way, also with the $2$ in the denominator, so this seems to be a common definition.


texts: Prediction interval for general non-linear regression

texts:
I have been working on a predictive model. With each prediction, we need to provide a score to express the confidence about our prediction. So I am looking at prediction interval (PI). In linear regression, I believe these can be obtained and well-documented. However, I am yet to find much reference for non-linear regression (such svr, gbr or other blackbox method for regression). Two methods that I have seen are given below:

1) Using bagging, we can generate many point prediction of each new data point, and then we get the interval from the distribution of these predictions around each new point.

2) using Quantile regression to get the upper and lower bound of the new point.

Personally, I do like the bagging method, although I don't feel very convinced. Hence, I am reaching out to the community to get a general opinion or some other ideas that I haven't seen so far.


texts: Predict User Demographics from location based social networks

texts:
I am currently working on an lbsn (localization-based social network) system and i need to predict the user's age and gender.

Every time a user enters a venue, the system creates a "check-in" with the user, the venue and the datetime.

Every venue is categorized using Foursquare Venue Categories.

The system generate a Weigthed Concept Hierarchy to represent the interest level between a user and a Venue Category.

Is it possible to predict the user's age and gender using the mentioned data?


texts: Is it possible to train for precision in Tensorflow?

texts: Can I train a binary classifier in Tensorflow to maximize precision?


texts: Different strategies for dealing with features with multiple values per sample in python machine learning models

texts:
I have a dataset which contains pregnancy, maternal, foetal and children data and I am developing a predictive machine learning model to predict adverse pregnancy outcomes.
The dataset contains mostly features with a single value per pregnancy, e.g. maternalObesity = ["Yes", "No]. However, I have some features that have multiple values per pregnancy, such as the foetal abdominal circumference and estimated foetal weight which have been recorded multiple times at different times during gestation (so each pregnancy will have between 1 and 26 observations for these features each), like so:
PregnancyID     gestationWeek    abdomcirc     maternalObesity
1               13               200           Yes
1               18               240           Yes
1               30               294           Yes
2               11               156           No
2               20               248           No

So in pregnancy 1, we can see that the abdominal circumference was recorded 3 times at weeks 13, 18 and 30.
All questions I have seen here which have addresses the issue of multiple values per sample have been about categorical features, like this and this. Here the suggested solution was to OneHotEncode the features. However, like I said, this does not apply in my case as I have continuous (float) variables.
I have spent the last few months attempting different methods to best handle these features such that I don't lose any valuable information. Simply adding these features into in my dataset will result in almost duplicate rows as the vast majority of my samples have single values (like in the table above.
Here are some of the different approaches I have considered to handle these features:

Derive statistical values from the features, like here. So I compute mean the maximum, minimum, variance, range, etc. of all the observations per pregnancy. However, the downfall with this approach is that the time at which the values are recorded is neglected. The time of the measurement may be significant as a higher abdominal circumference earlier in pregnancy may be more correlated with the adverse outcome I am trying to predict.

Summarise the measurements into a fixed number of features by grouping them into 3 trimester, like here. So I can group all measurements by 3 trimesters, and each feature would hold the maximum measurement recorded during that trimester.

So my dataset will look like this:
PregnancyID     MotherID    abdomCirc1st  abdomCirc2nd   abdomCirc3rd
1               1           200           315            350
2               2           156           248            NaN

This approach takes into account the time range of the measurement, but will result in a lot of NaNs in the new features, as many pregnancies do not have a measurement for each trimester. Also, the maximum may result in some statistical information being lost, unlike approach 1.

I initially thought about using a python list for these features. However, I do not know if a machine learning model can handle this data type, and again, the time each measurement was taken is neglected in this approach.

So my data will look something like this:
PregnancyID     maternalObesity    abdomcirc
1               Yes                [200, 240, 294]
2               No                 [156, 248]

In conclusion, I need some guidance as I have found a lack of examples and resources out there about this issue. So please advise what the best approach is in this case and if there are any detailed examples out there that address this issue I would appreciate it.


texts: Public dataset for account to account payments

texts:
I'm looking for a dataset which contains account to account payments (bank transfers). Ideally, this dataset would contain labeled data for transactions or accounts known to be victims of phishing attacks. In this scenario, the account holder enters and authenticates the transfer but has been tricked into making an undesired purchase or into sending the funds to an undesired recipient.
This could be a public dataset, or optionally I could collaborate confidentially on a private dataset and would sign the necessary confidentiality agreements.
I've looked at the repositories listed here already:
Publicly Available Datasets
I do know this credit card fraud dataset well, and it's the closest to what I'm searching for, but does not fill the requirements above: https://www.kaggle.com/mlg-ulb/creditcardfraud
For experts in this area, a more technical way to describe this fraud scenario is "authorized push payments".


texts: Feature Scaling of Training Set and Test Set

texts:
Suppose I want to use the Gradient Descent algorithm.
I have a training set and a test set and I want to do the feature scaling with mean normalization.

Should I use the same mean and variance for equivalent features in the 2 sets? Why?


texts: How to choose the order in which to split a decision tree?

texts:
I know that a decision tree recursively splits along each attribute, greedily minimizing the wrong classifications/deviance at each split.

But, what is the order in which the attributes are split?  In other words, for a regression tree in N dimensions, what determines which attribute gets split first?


texts: Good books or resources for better understanding mathematical notation used in Machine Learning / Deep Learning Papers

texts:
Can anybody recommend any books or resources that would be good for better understanding the mathematical notation that is often seen in ML/DL papers? Preferably with examples of how the notation can be "transcribed" into tensorflow code. I did Linear Algebra and Vector calculus in uni a long time ago but it is a long way back and I would like to get back up to speed with it.
This question I think is not what I am looking for because I would like to see an emphasis on how one can transcribe the math into code.


texts: I have recently read about Fine Tuning, and what I want to know is, when we are fine-tuning our model is it necessary to Freeze the model and train only the top part of the model and then unfreeze some layers and again train the model or one can directly begin by unfreezing some layers? Till now, I have read that one does not unfreeze the layers directly because then we risk losing the important features captured by the earlier layers


texts: Does the Bayesian MAP give a probability distribution over unseen data?

texts:
I'm working my way through the Bayesian world. So far I've understood that the MLE or the MPA are point estimates, therefore using such models just outputs one specific value and not a distribution.
Moreover, vanilla neural networks do in fact something like MLE, because minimizing the squared-loss or the cross-entropy is similar to finding parameters that maximize the likelihood. Furthermore, using neural networks with regularization is comparable to the MAP estimates, as the prior works like the penalty term in error functions.
However, I've found this work. It shows that the weights $W_{PLS}$ gained from a penalized least-squared are the same as the weights $W_{MAP}$ gained through Maximum a posterior:

However the paper says:

"The first two approaches result in similar predictions, although the
MAP Bayesian model does give a probability distribution for $t_*$ The
mean of this distribution is the same as that of the classical
predictor $y(x_*; W_{PLS})$, since $W_{PLS} = W_{MAP}$

What I don't get here is how can the MAP Bayesian give a probability distribution over $t_*$, when it is only a point estimate? Consider a neural network - a point estimate would mean some fixed weights so how can there be a output probability distribution? I thought that this is only achieved in the True Bayesian where we integrate out the unknown weights, therefore building something like the weight averaged of all outcomes, using all possible weights.


texts: Brain Connectogram or Connectivity Visualization

texts:
Can anybody please guide me how can I draw the following figure in my windows 10 environment, I mean is there some built-in online tool or software that I can use for this purpose?
In my case I have two average correlation matrices , one for healthy subjects and one for ADHD subjects and each matrix is 90 x 90 as I am using AAL brain tempalte.


texts: How to create a data generator for multi-output net?

texts:
I'm a beginner with Keras. I'm trying to train a model with two output using Keras.
The structure of my dataset is that.
--------------------------------------------------------------
filename | cat1    | cat2 | level1 | level2 | level3 | level4 |
--------------------------------------------------------------
1.jpg    | 1       | 0    | 0      |  0     | 0      |  0     |
2.jpg    | 0       | 1    | 0      |  1     | 0      |  0     |

....and so no

I've written the following model
    x = Input(shape=(320, 512,3), name="my model")
    x = res_net50(x, training=False)
    x = GlobalAveragePooling2D()(x)
    x = Dense(256, activation='relu')(x)
    x = BatchNormalization()(x)
    status = Dense(1, activation="sigmoid", name="status")(x) # cat1/cat2
    level = Dense(4, activation="softmax", name="level")(x)   # level1/level2/level3/level4
    model = Model(x, [status,level])

My dataset is little, so I decided to apply the augmentation technique. Until today, I have always applied this technique using the ImageDataGenerator class as follow
train_datagen = ImageDataGenerator(
 horizontal_flip=True,
 vertical_flip=True,
 rescale=1/255
)

validation_datagen = ImageDataGenerator(
 rescale=1/255
)

train_generator = train_datagen.flow_from_dataframe(
    train,
    directory="../dataset/",
    x_col = "filename",
    y_col = ['cat1', 'cat2', 'level1', 'level2', 'level3', 'level4'],
    class_mode="multi_output",
    target_size=(320,512),
)

validation_generator = validation_datagen.flow_from_dataframe(
    val,
    directory="../dataset/",
    x_col = "filename",
    y_col = ['cat1', 'cat2', 'level1', 'level2', 'level3', 'level4'],
    class_mode="multi_output",
    target_size=(320,512),
)

and used it on the fit method.
model.fit(
            train_generator,
            batch_size = 32,
            epochs = 50,
            callbacks = callbacks_list,
            validation_data = validation_generator)

From a similar question https://stackoverflow.com/questions/38972380/keras-how-to-use-fit-generator-with-multiple-outputs-of-different-type
I realized that I have to create one from scratch. However, I did not understand what procedure I have to follow because my case is a bit different.
Any help is appreciated including references to useful external documentation.


texts: How to feed multiple input data into model.fit() by using tf.data.Dataset?

texts:
Model :
model = vgg16(weights = 'imagenet', include_top=False)
cl1 = Dense(2, activation = 'softmax',name='class_1')(x)
cl2 = Dense(2, activation = 'softmax',name='class_2')(x)
model = Model(inputs=model.input, outputs= [cl1,cl2])

loss = ['categorical_crossentropy','categorical_crossentropy']
model.compile(optimizer=opt, loss=loss  , metrics=['categorical_accuracy'])

data_input_pipeline :
  train_generator=train_datagen.flow_from_dataframe()

  def custom_generator(generator):
    for img_batch, lb_batch in generator:
        img_batch_list = tf.data.Dataset.from_tensor_slices(img_batch)
        for img,img_lb in zip(img_batch,lb_batch):
              ** some code block to change img_lb into tuple , value of the key in tuple is in
               Tf.tensor shape(1,2)  **
              ** eg : updated_label_tuple ={class1:[1,0],class2: [0,1]}  **

        yield (updated_image_batch_tensor, updated_label_batch_list)

  train_ds =
  val_ds=

  model.fit(train_ds,
                    steps_per_epoch=200 ,
                    validation_data=val_ds,
                    validation_steps=40,
                    epochs=50,verbose=1)

Error :

check_types=check_types) File
"/home/samjith/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_core/python/data/util/nest.py",
line 299, in assert_shallow_structure
"Input has type: %s." % type(input_tree)) TypeError: If shallow structure is a sequence, input must also be a sequence. Input has
type: <class 'list'>.

Here i have to feed the data into 2 seperate dense layers which will perform 2 different tasks. I passed it as a tuple like mentioned here, but i got the above error ..
Is this right way to feed the data into two dense(2) layers ? How to solve this bug !!


texts:
Might sound like a very basic question, but in a detection problem with multiple objects (solved by regression) how does the weight update work such that regression equations of all classes are satisfied all at once. How is it that boxes of other objects do not interfere with the training of box weights of our object?
Help appreciated a lot. Thanking in advance! :)


texts: TensorFlow 2: Find MAE, RMSE for validation dataset in time-series LSTM

texts:
TensorFlow 2 "Time series forecasting tutorial" (https://www.tensorflow.org/tutorials/structured_data/time_series#recurrent_neural_network) gives an example of a LSTM multi-step prediction model that given a past history predicts a range of future values:

# Prepare data
train_data_multi = tf.data.Dataset.from_tensor_slices((x_train_multi, y_train_multi))
train_data_multi = train_data_multi.cache().shuffle(BUFFER_SIZE).batch(BATCH_SIZE).repeat()

val_data_multi = tf.data.Dataset.from_tensor_slices((x_val_multi, y_val_multi))
val_data_multi = val_data_multi.batch(BATCH_SIZE).repeat()

# Model with two LSTM layers
multi_step_model = tf.keras.models.Sequential()
multi_step_model.add(tf.keras.layers.LSTM(32,
                                      return_sequences=True,
                                      input_shape=x_train_multi.shape[-2:]))
multi_step_model.add(tf.keras.layers.LSTM(16, activation='relu'))
multi_step_model.add(tf.keras.layers.Dense(72))
multi_step_model.compile(optimizer=tf.keras.optimizers.RMSprop(clipvalue=1.0), loss='mae')

# Train model
multi_step_history = multi_step_model.fit(train_data_multi, epochs=EPOCHS,
                                      steps_per_epoch=EVALUATION_INTERVAL,
                                      validation_data=val_data_multi,
                                      validation_steps=50)

Tutorial also shows how to plot some values predicted by LSTM:

def multi_step_plot(history, true_future, prediction):
     ...
     ...

for x, y in val_data_multi.take(3):
     multi_step_plot(x[0], y[0], multi_step_model.predict(x)[0])

Question: How to iterate all x-s and all y-s (true values) in a validation dataset (val_data_multi)? I can not figure out how to access these values in tf.data.Dataset.from_tensor_slices. Also, how to get predictions from the model for all entries in validation dataset to be next used in calculation of mean  MAE and RMSE of the model?


texts: Regarding Training loss and validation loss

texts:
I am trying to use keras for multi label news classification. I am a beginner in machine learning so please bear with  me.

Here is my training loss vs validation loss diagram:

I understand that my model is overfitting.
On the test set, I have precision of 0.89 , recall of 0.82, f1score 0.83. This according to me is high precision and high recall. So this means a good classification right ?

Further, this is the parameter settings:

My question is why in the test set there is high precision and high recall but the model is overfitting in validation set. Isn't it odd? or am i missing something?

I would have uploaded my dataset. But the dataset excel file is 800 MB.

Note that I don't have a seperate training set, validation set and test set. I have a single file. On the first column are the articles and in the second columns are the labels.

I used sklearn.train_test_split twice so that the training set is 60% test set is 20% and validation set is 20%.


texts: K-Means clustering for mixed numeric and categorical data

texts:
My data set contains a number of numeric attributes and one categorical.
Say, NumericAttr1, NumericAttr2, ..., NumericAttrN, CategoricalAttr,
where CategoricalAttr takes one of three possible values: CategoricalAttrValue1, CategoricalAttrValue2 or CategoricalAttrValue3.
I'm using default k-means clustering algorithm implementation for Octave.
It works with numeric data only.
So my question: is it correct to split the categorical attribute CategoricalAttr into three numeric (binary) variables, like IsCategoricalAttrValue1, IsCategoricalAttrValue2, IsCategoricalAttrValue3 ?


texts: Effective Time Series Forecasting using Keras/LSTM

texts:
I am working on time series forecasting for an engineering component (turbo charger). I have dataset containing field data from sensors (=features) taken every day for different turbocharger for their entire life (~3 years). I am using following combination LSTM, Keras, RNN.

One of the critical factor is efficiency of the component. Given that is make little sense to continue to use the charger once the efficiency has declined below threshold. I want to forecast the efficiency of turbocharger (in following time steps): 1 week, 5 week, 15 week, 30 week and 52 week from the date of forecasting.

My field data (shown in dummy, in reality I have about 13 features) looks like:

import pandas as pd
import random
import numpy
df = pd.DataFrame(pd.date_range(start="2019-09-01", end="2019-09-30", freq='D', name='ds'))
df["Charger_ID"]=1
df["DATE_INT_STAMP"] = range(1,31)
df["sensor1"] = np.random.randint(1, 30, df.shape[0])
df["sensor2"] = np.random.randint(55, 89, df.shape[0])
df["sensor3"] = np.random.randint(21, 35, df.shape[0])
df["efficiency"] = np.random.randint(71, 90, df.shape[0])
df.head()

My confusion for effective use of LSTM/Keras/RNN regarding best practices are:

(1) For the kind of forecasting (1 week, 5 week, 15 week, 30 week and 52 week), what would be ideal sequence length/window over which LSTMs should look back? (Should it be 52 weeks?)

(2) I know that the sensor and efficiency data doesn’t change much every day, so, I am taking only weekly data while training the neural nets. Any trouble here?

(3) After I have trained and tested the model and is ready to be deployed for real use (to forecast for 5 specific time stamp). Should the input to model be a time series? or just a specific point on the time series from model should forecast?

(4) Given that this is multistep forecasting problem? Is there specific guideline that should be followed while converting time series to supervised problem? For now, I am taking input and output lag as 1?

(5) Lastly, I have used min_max_scaler to normalize training dataset and I am normalizing the test data using the parameters from the MinMax normalization applied on the training set. Should I save these parameters and use these for forecasting?


texts: How can I get a value of context vector in GPT?

texts:
I'm a newbie in NLP and I'm now stuck in GPT.
The question I'm struggling with is related to a term 'context vector'
It says in the following (sorry that the material provided is written in korean)
that U represents a context vector.
I searched for both terms 'context window' and 'context vector'
Now I understand that a context window means the number of words I will accept as a context of the token. I also searched for 'context vector' and what I found out is that it is calculated by multiplying value parameters of each token to its softmax (attention score)
(I hope I am right, feel free to tell me what I'm getting wrong)
My question is that if 'context vector' can be calculated by using attention mechanism, then how can it(U) be calculated in advance in GPT since a transformer block including a masked attention layer gets 'UW(e) + W(p)' as its input ?
I know this question might seems really non-essential and out of topics to many great experts here, but I guess it's just making me get stuck in a logical step to understand NLP.


texts: What is the best way to visualize range of data?

texts:
If I a have data like a bunch of classes where each class can have a different number of students, and each student has a grade (0-100), for example like this:
tdf = pd.DataFrame({'classes':["a", "a", "a", "b", "b", "b","b","b", "c", "c"],
                    'grades':[78, 28, 10, 3, 79, 91, 65, 51, 75, 89]})

What is the best way to visualize the range of grades within each class? I want to show that in class A, there is a range between 10 and 78, in B there is 3 and 91, in C there is 75 and 89. I was thinking of boxplot or violin plot but when I tried them I am not really sure. Is there such thing as a bar chart that can show range? Will that make sense?
I am not sure if this is the right place to ask, so here I am.


texts: Noise Elimination with majority vote filtering

texts:
I have a dataset with label noise which I wan't to clean with majority/consensus vote filtering. This will mean I will divide the data in K-Folds and train an ensemble model. Than using the predictions on the data I will remove rows, which are missclassified by most (majority voting) or all (consensus voting).

I have a few questions on which I can't find the answers elsewhere:

how to decide what models to use in the ensemble
the dataset is very imbalanced. Do I need to do upsampling in the majority voting?
do I do hyperparameter tuning in the different models, or just use standard settings?


texts: How to use Automated Labelling for documents?

texts: Let's say I have been given 1000 documents and 6 labels from someone. My job is to label each of these 1000 documents into 1 of the 6 labels which are words not numbers. How can I automate or semi-automate this process using data science??


texts: Gerrymandering - Geospatial optimization to maximize votes in R

texts:
I am looking for a practical guide/tutorial preferably in R to show how to do gerrymandering. (I was looking for it also in CRAN but didn't find such package) Gerrymandering is the manipulation of the boundaries of electorial districts in order to gain political advantage for one party. If there is an analogy/similar process in another area it would be also interesting.

I have detailed historical  election results. I can see the results from each "voting office" and also the geographical area they cover. These results are aggregated to electorial district level. I'd like to see how moving voting offices to different electorial districts can influence the results of the election and how this process can be optimized for one party. Certainly there should be some constraints not to create really weird shaped districts at the end.


texts: How to have a DNN output a classification for each user at once?

texts:
I have a Reinforcement Learning environment with an agent that allocates power values to different users.
To do so, I have thought of implementing a deep neural network like the one shown in the picture below:

The input layer will receive information about all the users at once (like their Signal to Noise Ratio or SNR etc).
Each user will start with an initial average power value.
The DNN must decide whether to add/substract to/from the initial value of the power of each user, or do nothing, hence having groups of neurons for each user, each group has 3 neurons, each neuron represents one of the 3 options (add/substract/do nothing).
The output layer uses a Sigmoid activation function, and for each group the action with the best prediction will be applied.
The DNN will receive a reward that depends on how many users had their QoS requirements satisfied.
I am new to Deep Learning and I was wondering if there is any better way to approach this problem.


texts: Trying to figure out which book to tackle first

texts:
I'm currently at my 4th and last year of my Applied Maths degree and just recently got an internship at a startup. During this period I've been reading and getting deeper and deeper into studying machine learning. And so I've stumbled upon two interesting books:
1- "The elements of statistical learning", Jerome H. Friedman, Robert Tibshirani, and Trevor Hastie
2-"Understanding machine learning (From theory to algorithms)", Shai Ben-David, Shai Shalev-Shwartz
I intend to read both books but just wanted to hear from someone more experienced in this field which one i should focus on first, and each books strenghts and weaknesses.


texts: Is there some model-based variation of the Q-Learning algorithm which learns on a 3D SxAxS' table instead of a 2D SxA table?

texts:
Q-learning works with a 2D SxA table of Q values, where S is the current state and A is the action taken. Is there some model-based variant of Q-learning (or SARSA) that uses a 3D SxAxS' table to store Q values, where S' is the resulting state?

A similar table may be used to learn models in discrete environments, but in this case you don't store Q values, but counts.

I couldn't find anything like this, either because I don't know the correct terms to search for, or because it does not exist. It doesn't matter to me if it is useful or not, I just need to know if it exists, and if it exists, how is it called.


texts: Variance of product of a matrix and vector using Python

texts:
I am trying to calculate the variance of the product of matrix A and vector b. As it was mathematically discussed on another post https://math.stackexchange.com/questions/2365166/what-is-the-variance-of-a-constant-matrix-times-a-random-vector, I have tried to do the calculation in python.
To recap, we have
\begin{equation}
Y = \bf{Ab}
\end{equation}
where \begin{equation}
\bf{b} \sim N(\mu_b, \bf{I}\sigma_b^2)
\end{equation}
Therefore
\begin{equation}
Var(Y) = \bf{A}\sigma_b^2\bf{A}^T
\end{equation}
I have tried this code with
import numpy as np
import statistics

A = np.arange(12).reshape(4, 3)   # some 4 by 3 array
b = np.array([2,4,5])              # some 3 by 1 vector

Y = np.dot(A,b)
statistics.variance(Y)
>> 1815

Vb = np.dot(np.identity(b.shape[0]), statistics.variance(b))
np.dot(A.dot(Vb), np.transpose(A))
>> array([[ 10.,  28.,  46.,  64.],
       [ 28., 100., 172., 244.],
       [ 46., 172., 298., 424.],
       [ 64., 244., 424., 604.]])

However, the statistic.variance(Y) outputs a different result compared to np.dot(A.dot(Vb), np.transpose(A)).
I wonder how should I do it properly in python?
Your help is really appreciated!


texts: Is there a person class in ImageNet? Are there any classes related to humans?

texts: If I look at one of the many sources for the Imagenet classes on the Internet I cannot find a single class related to human beings (and no, harvestman is not someone who harvests, but it's what I knew as a daddy longlegs, a kind of spider :-). How is that possible? I would have at least expected a person class, and even something more specific such as man, woman, toddler, etc. Nothing of the sort. Why? Did Fei-Fei Li and her team make a conscious choice not to have people images in the database? Am I looking at the wrong file? For the sake of the question, we can consider the ImageNet versions from 2014 onwards.


texts:
I am reading gail implementation code in openai baselines. they compute bernoulli entropy as one of the loss in adversary network loss function.

In their code, they implement bernoulli entropy as this:

def logsigmoid(a):
'''Equivalent to tf.log(tf.sigmoid(a))'''
return -tf.nn.softplus(-a)

def logit_bernoulli_entropy(logits):
ent = (1.-tf.nn.sigmoid(logits))*logits - logsigmoid(logits)
return ent

also there is a reference of another openai implementation, it's the same code but I can't see any explanation to it.

I checked that the equation to compute bernoulli entropy is:

$ -p\log p - (1-p)\log(1-p)$

I think the later equation is the right way to compute bernoulli entropy, but the first one should be right too as it's written in openai's implementationn. I can't see any similarity, is there any relationship between these two expression?


texts: Obtain learning curve of Gradient Boosted Tree model in PySpark

texts: Currently there seems to be no method in PySpark of checkpointing the performance of a model at each gradient update. Is there a way to get the performance of a model at each gradient update so that a learning curve can be plotted?


texts: How to convert 200 bag of word features to several continuous features

texts:
I am using one text dataset for regression task. The baseline regression method (name concealed) that I employ will work well if all the input features are continuous.  However, the text dataset that I am considering has 200 bag-of-word features that are binary (0, or 1), i.e the first feature will be 1 if the associated word appearing on the text and 0 otherwise. I am just curious how can I convert those 200 binary to one or several continuous features such that my regressor can work with.
Thanks


texts: Estimating class prevalence in unlabelled data after predicting labels with a binary classifier

texts:
I'm looking to get an estimate of the prevalence of 1's (i.e. the rate of positive labels) in a very large dataset that I have. However, I am hoping to report this percentage as a 95% credible interval instead of as an exact estimate of rate, taking into account the model uncertainties.

These are the steps I'm hoping to perform:

Train a binary classifier on labelled training data.
Use a labelled test set to estimate the specificity and sensitivity of the classifier.
Use the classifier to predict the label for the unlabelled records in the dataset.
Obviously I could get an exact prevalence estimate by simply calculating the mean of the predicted outputs. But this is where I'm hoping to implement an approach for reporting the prevalence estimate as an interval.

So my question is: Is there a best-practice approach to doing this? I found this study which trains a binary classifier and then uses a Bayesian prevalence model to report the prevalence as a 95% confidence interval by incorporating the uncertainty associated with the model specificity and sensitivity. However, I'm having trouble understanding exactly what they did here. I'm also not finding many others who have done something similar. So, any suggestions for a reliable approach I could take to do this would be greatly appreciated.

Thanks in advance!


texts:
Suppose I have a dataset with $M$ observations of dimension $N$.

I've seen many people use Kohonen networks (or self-organizing maps, SOMs) to do this task. In this approach, the neurons have weights of the same dimension of the observations, $N$. The idea is to train the network so that each node $w_i$ becomes a centroid of a cluster, and thus identify the clusters with the neurons of the network. As we don't know the number of clusters a priori, one should establish a large number of neurons, $W$ such that $W<< M$, and then identify the nodes that represent the centroids (checking which ones are located in more dense areas and other criteria similar to that one).

What I don't understand is why one would do this. I could just say that I have $M$ possible centroids (each datapoint) and then use the discriminating criteria to check which ones are the actual centroids. In this approach there are $M$ centroid candidates and in the SOM one, $W$. But in both cases the dimension is $N$. Why is then using a SOM favorable? I believe I would want to have more data when performing clustering, so I think that having $M$ samples instead of $W$ would be better. Also, the dimensionality of the problem stays the same, so no advantage there. On top of it, it would be much faster to do it without the SOM, as training it may take a while, but the output (i.e. the values of the neuron weights) would just be pretty similar to a subset of the original dataset, which could be obtained just by identifying possible centroids from the $M$ observations.

Why is it then that SOMs are used for clustering?


texts: Measure of correlation for term frequency

texts:
I'm trying to write a framework to compare a set of labels such as (for a sample of 5 yes/no answers to a question)  [0, 1, 1, 1, 0] to a series of features to determine correlation.  For numerical non-sparse features, like "number of words" or "average word length", I know I can use a variance-covariance matrix and get a sense for whether or not "number of words" or "average word length" is an informative feature for a model to answer the question.

I'd like to be able to do the same thing for term frequency (let's say using CountVectorizer in scikit-learn), but the resultant covariance matrix will be rather large and will only indicate whether or not that particular term is an informative feature.  How do I get some kind of "collapsed" or "aggregate" measure of correlation?  Is this even possible?


texts: Functions in scikit that detect outliers automatically?

texts: I know a way to visualize outliers is to make a box plot, but wanted to know if scikit had any quick ways to detect outliers for each variable?


texts: How to analyze which site has most numbers

texts:
I am trying to determine which site in our organization is in greater need of upgrades to SEP 12, so when I run a query to count, I get these type of numbers

Group       Windows_SEP_11  Mac_SEP_11  Windows_SEP_12  Mac_SEP_12
Arizona\A   417                  29              219         6
Arizona\B   380                  20              282        15
Arizona\C   340                  30              383        507
Arizona\D   310                  104             186        857
Arizona\E   307                  74              403        243
Arizona\F   285                  171             522        14
Arizona\G   269                  1               559        41

However, when I find percentages, I get these numbers

Group          Win_Sep_11_%   Mac_SEP_11_%  Windows_SEP_12_%    Mac_SEP_12_%
Boston/Site 1   100               0                0               0
Boston/Site 2   100               0                0               0
Boston/Site 3   94                0                0               5

And obviously, percentage isn't good indicator because Boston/Site 1 has only 3 computers, Boston/Site 2 only has 4 computers, etc.

What is the best way to analyze data? I ultimately need a visual of sites that have

many computers, and
a great need for upgrades to SEP 12, i.e. if there are more computers with SEP 11 than SEP 12.

Please point me in the right direction.


texts: LSTM input and output for sentiment analysis

texts:
I'm studying this LSTM network:
https://www.kaggle.com/paoloripamonti/twitter-sentiment-analysis

model = Sequential()
model.add(embedding_layer)
model.add(Dropout(0.5))
model.add(LSTM(100, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(1, activation='sigmoid'))

model.summary()

I understand the input part of the embedding layer. Each word get's a unique vector that represents the meaning of the word.

The drop out will deactivate neurons.
So the input for the LSTM model is the vocabulary where each token is represented by a vector.

I understand the workflow in an LSTM model. But what exactly does it do with the input? Give it a score by learning?
And what is the output of the lstm?

Here is the summary:


texts: Storing outputs from convolutional layers

texts:
I am working on a CNN architecture over video clips that attempts to separate temporal and spatial features, inspired by this paper. The relevant architecture is on Figure 5(b).

Here is my idea: I want to take each frame and feed it through two 2D convolutional layers. After I feed each frame through these layers, I want to take all the frames and feed them into a 3D convolutional layer as one input, so that I can extract temporal features. However, I don't know how to save each output of the two 2D convolutional layers in Tensorflow. Any advice would be helpful.


texts: Calculate the Convolutional Autoencoder sizes - Conv1D

texts:
I'm approaching the Conv1D for the first time and I do not understand how to calculate the parameters in each layer. I have an input of (3000, 10, 30), but I decided to use a batch=10, so it becomes (10, 10, 30). Since I'm creating an autoencoder I need an output of the same size. This is my code:
model = tf.keras.Sequential(
    [
        tf.keras.layers.Conv1D(
            filters=128, kernel_size=8, padding="same", strides=3, activation="relu"
        ),
        tf.keras.layers.Dropout(rate=0.2),
        tf.keras.layers.Conv1D(
            filters=64, kernel_size=8, padding="same", strides=1, activation="relu"
        ),
        tf.keras.layers.Dropout(rate=0.2),
        tf.keras.layers.Conv1D(
            filters=32, kernel_size=8, padding="same", strides=1, activation="relu"
        ),
        tf.keras.layers.Conv1DTranspose(
            filters=32, kernel_size=8, padding="same", strides=3, activation="relu"
        ),
        tf.keras.layers.Dropout(rate=0.2),
        tf.keras.layers.Conv1DTranspose(
            filters=64, kernel_size=8, padding="same", strides=1, activation="relu"
        ),
        tf.keras.layers.Dropout(rate=0.2),
        tf.keras.layers.Conv1DTranspose(
            filters=128, kernel_size=8, padding="same", strides=1, activation="relu"
        ),
        tf.keras.layers.Conv1DTranspose(filters=30, kernel_size=8, padding="same"),
    ]
)

The problem is that when I run it I get the error:

ValueError: Dimensions must be equal, but are 12 and 10 for '{{node mean_squared_error/SquaredDifference}} = SquaredDifference[T=DT_FLOAT](sequential/conv1d_transpose_3/BiasAdd, IteratorGetNext:1)' with input shapes: [10,12,30], [10,10,30].

Does the formula changes when using a Conv1D layer? At the moment I'm calculating it using ((shape of width of the filter * shape of height of the filter * number of filters in the previous layer+1)*number of filters), but as you can see it doesn't work.


texts:
I want to get into machine learning. I've been in information security for the last 10 years so i have an IT background.

Where is the best place to start:
Can anyone recommend a good book? And also a platform i can use to practice (preferably free)
Also if there are any online courses someone could recommend that would be great.

I looked into AWS's offering of machine learning but that is not included in the free tier.

Any help/advice would be much appreciated.

Thank You.


texts: I have a question regarding my machine learning lecture where we had to decide whether $$K(x,y)=x_1y_1-x_2y_2$$ is a valid kernel (e.g. for a SVM). My intuition would say that it is a valid kernel since we can display it with: $$\Phi(x)=(x_1, ix_2)\implies K(x,y)=\Phi(x)\Phi(y)$$ with $i$ being the imaginary number. Is that right?


texts:
If I analyse a random forest in python with scikit I do:

target = "time"

dataIn = data[features + [target]]

# Splitting data into x % training data and Hyperparameter.TestSize % test data
X, y = dataIn[dataIn.columns.difference([target])], dataIn[target]

X_train, X_test, y_train, y_test = model_selection.train_test_split(X, y,
                                                                    test_size=TestSize,
                                                                    random_state=42)

If I do a gradient boosted random forest in R I do:

#split data into training and test data sets
trainIndex = createDataPartition(daten$target, p=0.9, list=FALSE, times=1)
train = daten[trainIndex,]
test = daten[-trainIndex,]
rownames(train) <- NULL
rownames(test) <- NULL

gbmTraining <- gbm(train$target~., data=train, distribution = "gaussian", verbose = TRUE,
                   n.trees = 100,
                   cv.folds = 4,
                   n.cores = NULL,
                   shrinkage = 0.3,
                   interaction.depth = 7,
                   n.minobsinnode = 15)

As you see in the scikit implementation I have the target not in the data set. There will be created a new dataframe that has only the target feature as column.

In the gbm R implementation instead the dataframe train contains my target and I communicate to it with train$target.

Therefore I want to ask if you split the data in train and test parts in machine learning algorithms in general (I mean I am comparing two different models), do the target has to be a column in the train data set or do the target has to be a dataframe for it's own?


texts: Brownian motion in models for virus spread?

texts:
Was reading a Washington Post article "Why outbreaks like coronavirus
spread exponentially, and how to flatten the curve” and it looked like they were using Brownian Motion.

(Can't directly link the graphic, but it's the third graphic in the article, showing particles bouncing around, but I've included a screenshot:)

If this is the case, how do particles in a fluid medium resemble interactions in a human population?


texts: Learning a logical function with a 2 layer BDN network - manual weight setting rule question?

texts:
So I am trying to construct a 2-layer network of binary decision neurons as proposed by McCullough and Pitts (1943) to learn a logical function (a composition of AND's and OR's) such as:
$((\neg x_1\land x_3) \lor (\neg x_2 \land x_1 \land x_3))$

I have gotten as far as attempting to construct a 2-layer network which I believe should be able to learn this function, each neuron $h_1$ through $h_3$ performing a single logical operation.

However my lack of understanding in both logic and BDNs as a whole has led me to believe that my network is incorrect when considering the weight setting rules proposed by McCullough Pitts (j is neuron index, k is input index):

$w_{jk} = -1$ if $F(x_k)=\neg x_k $

and

$w_{jk} = 1$ if $F(x_k)=x_k $

As far as I understood it, you set the weight for each input to -1 if that input is negated in the neuron's formula and 1 otherwise. I was able to calculate these for the first layer with ease, but got thrown by a hole in my knowledge of how the second layer works. I assumed that $h_3$ takes $y_1$ and $y_2$ as input, as opposed to an $x_k$, so I don't see how the above rule can be applied.

I was hoping maybe someone with a stronger understanding of neural networks could perhaps explain to me how the weight setting rule is applied to the second layer, I currently assumed it meant $w=1$ if $F(y_i)=y_i$ and visa versa, but I believe this is wrong.


texts: time series prediction with several independent variables using RNNs and Keras

texts: I came across this excellent question and answer by Adam Sypniewski. Here method of using RNNs (e.g. LSTMs and GRUs) for predicting time series where the dependent variable depends on several independent variables. Just wondering, is anyone aware of some Keras code for this?


texts:
I am going through this course. The professor is talking about the logistic regression cost function:

$$
P(y|x) = \hat y^y (1- \hat y)^{(1-y)}
$$

Taking log on both sides provides:

$$
\begin{align}
log[P(y|x)] &= log[\hat y^y (1- \hat y)^{(1-y)}]\\
&= - L(\hat y,y)
\end{align}
$$

Why does the logistic regression cost function need to be the negative of log?


texts: Is my model overfitting? The validation loss keeps on fluctuating

texts:
I have trained a 4 layer neural network

model = Sequential()

#get number of columns in training data
n_cols = X_train.shape[1]

#add model layers
model.add(Dense(8, activation='relu', input_shape=(n_cols,)))
model.add(Dense(8, activation='relu'))
model.add(Dropout(rate = 0.05))
model.add(Dense(8, activation='relu'))

model.add(Dense(1))
#adam = optimizers.Adam(lr=0.0001, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)
model.compile(optimizer='adam', loss='mae')
history = model.fit(X_train, y_train, epochs= 200, validation_split=0.2, batch_size=128)

When I plot the graph between train and validation loss the graph seems to be like
The validation loss is fluctuating. Am I doing it right?


texts:
This is my approach

svm = cv2.ml.SVM_create()
svm.setType(cv2.ml.SVM_C_SVC)
svm.setKernel(cv2.ml.SVM_RBF)
svm.trainAuto(data["encodings"], cv2.ml.ROW_SAMPLE,data["names"])

where I get encodings using HOG and names from:

for (i, imagePath) in enumerate(imagePaths):
    # extract the person name from the image path
    print("[INFO] processing image {}/{}".format(i + 1, len(imagePaths)))
    name = imagePath.split(os.path.sep)[-2]

but when I train

  'samples is not a numpy array, neither a scalar'

this error appears in the line:

svm.trainAuto(data["encodings"], cv2.ml.ROW_SAMPLE,data["names"])

I think I need to convert the names to a numpy array.

How can I do that?


texts:
I hope I can make my question clear.

So I'm working on a machine learning project just for the sake of it (3rd year undergrad in CS) over our summer vacation.

So I'd like to recognize letters from common fonts. I downloaded my data set from chars74k dataset

http://www.ee.surrey.ac.uk/CVSSP/demos/chars74k/ (link to the dataset)

What I've done up until now:

I cleaned up the images a bit, resized the images, cleaned them from noise, resized them to be 32x32 etc. (also deleted some poor fonts that were not correct such as capital letters categorized as small letters etc).

Now to my problem: I'm trying to find a model that has a high test accuracy (as far as im concerned I don't mind overfitting the data because I only want to deal with fonts, so if I can categorize these fonts correctly I'll probably be alright on the majority of common fonts)

In any case the first model ive tried is KNN over HoG features of the letters, playing around a bit with stride and number of bins the best Ive managed so far is 87.3% accuracy.

So I'd like some advice on what other ML models would work, I'm looking in to CNNs and SVMs right now but can't find any literature on useful architectures. So can some expert offer me any advice?


texts:
I have a list of transactions of bus route from place to place, I don't have any target variable here. I was asked to give meaningful insights from the data, what can I do here?

I cleaned the data, replaced missing values in distance, price. Doing only descriptive analytics, what predictive analytics can we give here?

Here is a sample of my data,


texts: How and where to set weights in case of imbalanced cost sensitive learning in machine learning?

texts:
I confront with a binary classification machine learning task which is both slightly imbalanced and cost sensitive. I wonder what (and where in the modeling pipeline, say, in sklearn) is the best way to take all these considerations into account.
Class proportionality: positive: 0.25% negative: 0.75%. This could be addressed with sklearn.utils.class_weigh.compute_class_weight:
class_weights = compute_class_weight(y=y, class_weight='balanced')

OK, but this is only for rebalancing proportionalty, I should take misclassification cost into consideration as well. Let's say that this is 10* larger in case of false negatives than false positives, so I guess that I should still increase the weights in class_weights accordingly by upweighting positives further by 10, right?
But there is another point in the pipeline where I could take care of this, namely in the evaluation metrics with F-beta for example with upweighted recall (F2, for instance). Does it have the same effect? Should I pick one method (F-beta for evaluation OR upweighting classes) or both of them simultaneously?.
Additionally, in case I upweight my classes with compute_class_weight(), I assume that no further class distribution should be taken into consideration downstream (so when I use RandomForestClassifier(), class_weight hyperparameter shouldn't be ='balanced', again, because this would further distort the weights proportionality that is already set before. Is this correct?


texts: where does the database created in spark go?

texts:
I am following this tutorial:https://docs.microsoft.com/en-us/learn/modules/intro-to-azure-databricks/4-using-notebooks

In this tutorial we create a database like this:

%sql

CREATE DATABASE IF NOT EXISTS Databricks;
USE Databricks;

CREATE TABLE IF NOT EXISTS AirlineFlight
USING CSV
OPTIONS (
  header="true",
  delimiter=",",
  inferSchema="true",
  path="dbfs:/mnt/training/asa/flights/small.csv"
);

CACHE TABLE AirlineFlight;

SELECT * FROM AirlineFlight;

Where is this database created? moreover there is a question asked

Question: Which of the following are good applications for Apache Spark? (Select all that apply.)

Querying, exploring, and analyzing very large files and data sets
Joining data lakes
Machine learning and predictive analytics
Processing streaming data
Graph analytics
Overnight batch processing of very large files
Updating individual records in a database
Answer: All but #7. Apache Spark uses SQL to read and performs analysis on large files, but it is not a Database.

If we can create a database using spark then why can't we change its records too


texts: Ordinal Regression to understand Google Rankings

texts:
I've made a dataset of search engine rankings for web pages versus a host of on-page factors (such as the amount of words on the page or the lenth of the  tag) and I would like to try and build a model to understand which of these on-page factors influences rankings the most (if at all).
I want to us the Mord Package  to run an ordinal regression as it does it out of the box and it should be interesting to optimise and find out the weights of coefficients.
Does this sound like a reasonable approach?


texts: can we have different features for different hyperplanes in SVM?

texts:
is it possible if we have different features for different classes of svm?

For example one of the hyperplane:

$$w_1\cdot \text{age}+ w_2 \cdot \text{ trip duration} +w_3 \cdot \text{ income}$$

and the other hyperplane be

$$w_1\cdot \text{cost}+ w_2 \cdot \text{trip duration} +w_3 \cdot \text{purpose of trip}$$

the other one

$$w_1\cdot \text{distance}+ w_2 \cdot \text{trip duration} +w_3 \cdot 0?$$

Does it make sense?


texts: Are there any classification schemes like a support vector machine that use axis-aligned boxes instead of hyperplanes?  I have a dataset consisting of about a billion points in 9 dimensions, which are divided into two classes.  I would like a classification rule that gives me a collection of disjoint boxes, such that the rule for classifying query point $x$ is to say "if $x$ is in any of the boxes, it's class 1, and otherwise, it's class 2".  Is this a thing?


texts: Can we call a CMS like Drupal Machine Learning?

texts:
Drupal process a page and created run time PHP files, Twig files, sometimes CSS/JS as well, from the fragments we have already created based on it's logic.
Since machine learning does some similar thing to act on the data and use own logic to produce result, can we can CMS to be bit of Machine Learning capabilities?


texts: Given a 12x12 binary image (only black and white pixels) what is its dimensionality? And how can I define dimensionality of a data space?

texts: Suppose I have a grid 12x12 of pixels that can be only black or white. I can't understand if the dimensionality is 2 or 3. I mean... Is dimension given by 12x12 or 12x12x2 ?


texts: General strategy for imbalanced, semi-supervised, sparse problem

texts:
I am looking for some general advice on where to start with this problem.  There are 350 sparse (low positive integer) features.  I have 2000 positives, 1000 negatives, and infinite unlabeled data, where the estimated true positive rate in the unlabeled data is ~1% (but this isn't known for certain).  The goal is to predict the probability of an example being positive.

How would you approach the problem if you had roughly 40 hours to attain the best possible model?

What about if you had more time?


texts: Default parameters for decision trees give better results than parameters optimised using GridsearchCV

texts:
I am using Gridsearch for a DecisionTreeClassifier predicting a binary outcome. When I run fit and predict with default parameters, I get the following results:

Accuracy: 0.9602242115860793
F1: 0.9581087077004674

Then I try GridsearchCV:

from sklearn.model_selection import GridSearchCV

param_grid = {"criterion": ["gini", "entropy"],
              "min_samples_split": [2, 10],
              "max_depth": [2, 5, 10]
              }

grid = GridSearchCV(dtc, param_grid, cv=3, scoring='neg_mean_squared_error')

grid.fit(X_train, y_train.values.ravel())

y_pred_class = grid.predict(X_test)

When I check the results in y_pred, they only contain one class (0) and thus I get a warning when I try to look at F1:

  site-packages\sklearn\metrics\classification.py:1135:
  UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in
  labels with no predicted samples.   'precision', 'predicted', average,
  warn_for)

Could anyone please suggest what could be the issue here and why the best parameters give the same prediction for the whole set?

Answer: With that many features in my dataset, restricting max depth and min samples split with the specified values could not give adequate results. Adding None and increasing the range fixed the issue.


texts: Item-to-Item similarity: is weighted sum the most popular approach?

texts:
In content-based filtering (CBF) recommenders, when there are is no user profile, similar items are recommended an item that a user is currently inspecting. For instance, if you are looking for a comedy movie, other comedies might be recommended.
In the literature I found that often such recommenders use linear combination (i.e., weighted sums) of distances between items features to measure similarity between items.
I wonder how common the approach with weighted sum is for CBF?
Is industry using rather linear classifiers to measure similarity between items?


texts: Why the validation accuracy is constant for all the pre-trained models while applying on medical datasets with .czi format?

texts:
#import all the libraries
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix,classification_report
import os
import sklearn.utils

path='/content/drive/My Drive/Drug/spheroid_imageset/untreated/training_set'

X=np.load(path+'/X.npy')
Y=np.load(path+'/Y.npy')

X=X.reshape(-1,224,224,1)

X,Y = sklearn.utils.shuffle(X,Y)

tf.keras.callbacks.EarlyStopping(
    # monitor="val_loss",
    # min_delta=0,
    patience=3,
    verbose=0,
    mode="auto",
    baseline=None,
    restore_best_weights=False,
)

image_size=224

base_model = tf.keras.applications.MobileNet(weights=None,include_top=False,input_shape=(image_size,image_size,1))
base_model.trainable = False

x = base_model.output
x = tf.keras.layers.GlobalAveragePooling2D()(x)
x = tf.keras.layers.Dropout(0.50)(x)
x = tf.keras.layers.Dense(16,activation='relu')(x)
x = tf.keras.layers.Dense(3,activation="softmax")(x)
model = tf.keras.Model(inputs=base_model.input, outputs=x)
model.compile(optimizer='adam',loss='sparse_categorical_crossentropy',metrics=['accuracy'])

history=model.fit(X,Y,epochs=50,validation_split=0.25,callbacks=[callback])


texts: Neural Network written in Kotlin working for simple math problems but not for MNIST classification

texts:
I'm building a neural network in Kotlin while reading the book "Neural Networks and Deep Learning" from Michael Nielsen.
At the moment the network uses: sigmoid neurons, backpropagation, stochastic gradient descent, $1 \over \sqrt {n_{in}}$ standard deviation for weight initialization, L2 regularization, and cross-entropy. I also created a plotter to monitor the learning process.
Here is how the network behaves with a simple problem, finding the bigger of two numbers:

Pretty good, we have a peak of 99.99%.
I can't understand why the cost is increasing instead of decreasing, but still the accuracy is good.
Things get way worse with MNIST classification; the net is basically giving random answers:

I wonder what can change the behaviour of the network this much.
I checked the MNIST data, and it seems all right.
What problem can cause this kind of learning absence? The algorithm is broken? The hyper-parameters are wrong?
All help is appreciated.


texts: What Machine Learning Algorithm could I use to determine some measure in a date?

texts:
I am getting stuck with this problem. Let's say that we have the next information.

CustomerID: 1, Date: 3/2/2018, Quantity: 3, Total: 390.78, Min: 130.26, Max: 130.26

We want to determine given a day, month and year what will be the total summation of sales until the last day of that month, I'm using Microsoft Azure Machine Learning Studio and I modified the data a little bit (I know a little bit of Python and R), for a day we could have the next data.

Date: 3/2/2018, TotalSalesToday: 4023.45, FirstToToday: 1322.92 TargetValue: 42611.27

where TargetValue is the total summation of sales until the last day of the month (or the value that we want to predict), FirstToToday the Summation of the sales from the first day up to that day, TotalSalesToday the total summation of the sales of that day. There are some columns that we can find or generate given the day like RemainingWorkDays, RemainingHolidays, RemainingNonWorkDays, etc. And maybe 31 columns that we can make telling the ML what is the summation of all the days before -1, all the days before -2, and so on.

I did an experiment on Microsoft Azure Machine Learning Studio and it's giving a 100% coefficient of determination (I was using Boosted Decision Tree Regresion and Tunel Model Hyperparameters), I think that's because the ML knows that in a given month the TargetValue doesn't change it's value, so it does something like if(month == 2) PredictValue = 42611.27, what I can do? When testing this ML let's suppose that in the first day we got a TotalSalesToday: 1000000 so my ML returns 50000, and obviously this is not a logical and coherent answer regarding this value (1000000).

Is there something I need to change in the data? What do we need in order to make the ML gives at least a coherent answer? Is there something I forget?

Thanks in advance!! :)


texts: Score remains same during hyper parameter tuning

texts:
My model:

    model = Sequential()
    model.add(Dense(128, activation='relu', input_dim=n_input_1))
    model.add(Dense(64, activation='relu'))
    #model.add(Dense(32, activation='relu'))
    #model.add(Dense(16, activation='relu'))
    model.add(Dense(1))
    model.compile(optimizer='adam', loss='mse',metrics=['mse'])

Now, I am doing hyper parameter tuning, but it is showing the same for every possible result:

Best: -61101.514139 using {'batch_size': 10, 'epochs': 2}
-61101.514139 (25108.783936) with: {'batch_size': 10, 'epochs': 2}
-61101.514139 (25108.783936) with: {'batch_size': 10, 'epochs': 4}
-61101.514139 (25108.783936) with: {'batch_size': 10, 'epochs': 5}
-61101.514139 (25108.783936) with: {'batch_size': 10, 'epochs': 10}
-61101.514139 (25108.783936) with: {'batch_size': 10, 'epochs': 15}
-61101.514139 (25108.783936) with: {'batch_size': 20, 'epochs': 2}
-61101.514139 (25108.783936) with: {'batch_size': 20, 'epochs': 4}
-61101.514139 (25108.783936) with: {'batch_size': 20, 'epochs': 5}
-61101.514139 (25108.783936) with: {'batch_size': 20, 'epochs': 10}
-61101.514139 (25108.783936) with: {'batch_size': 20, 'epochs': 15}
-61101.514139 (25108.783936) with: {'batch_size': 30, 'epochs': 2}
-61101.514139 (25108.783936) with: {'batch_size': 30, 'epochs': 4}
-61101.514139 (25108.783936) with: {'batch_size': 30, 'epochs': 5}
-61101.514139 (25108.783936) with: {'batch_size': 30, 'epochs': 10}
-61101.514139 (25108.783936) with: {'batch_size': 30, 'epochs': 15}

This is the first time I have done hyper parameter tuning before and this has stumped me. I can provide additional details if needed. What is a possible reason for this behavior?

I am doing time series forecasting using MLP. I have used 'neg_mean_absolute_error as the scoring function in gridsearchCV.

Edit: this is what Im running:

from sklearn.model_selection import GridSearchCV
# fix random seed for reproducibility
seed = 7
np.random.seed(seed)

# define the grid search parameters
model = KerasClassifier(build_fn=create_model, verbose=1)
batch_size = [10,20,2000]
epochs = [2,4,5,10, 25]
param_grid = dict(batch_size=batch_size, epochs=epochs)
grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1, cv=3,scoring='neg_mean_squared_error')
grid_result = grid.fit(scaled_train,scaled_train_y)
# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
    print("%f (%f) with: %r" % (mean, stdev, param))


texts:
I have trained and validated my LSTM and I would like to deploy it. So, I know that we can save and load the Sequential object of Keras (I am working with Keras as you can guess). I thus implemented a code using these functions.
But, I would like to know whether I must train my model with all the available data (training + test) or just on the training set as I did during my study.
Many tutorials talk about how to train a model but not so many are clear about how to deploy a model.
I would like to know what is common to do, knowing that I am doing Time Series Forecasting which is a specific problem.


texts: Understanding the significance of LeNet-5 w/ MNIST data set

texts:
I'm beginning to learn about conv nets and started with what I understand to be one of the seminal works: LeNet-5. However, my limited experimentation doesn't seem to show any advantage over a single hidden layer fully-connected network. I'd like to understand what's going on.

For reference, the LeNet-5 paper is here: http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf

The Fully-Connected Network

The MNIST data set has 60k training examples of 28x28 pixels each, and 10k in the test (validation) set. My fully-connected network has this topology:

Layer 0 (input): 28*28 = 784 inputs
Layer 1 (hidden): 100 units (ReLU and sigmoid activation were tested)
Layer 2 (output): 10 units (softmax)

Cross-entropy loss was used and stochastic gradient descent with a batch size of 64 and 10,000 iterations was used for training using custom code I wrote myself.

LeNet-5

I used the LeNet-5 example here. It's not identical to the original network in that it uses ReLU activations, max pooling instead of average pooling, and has more connections than the original implementation in layer 3 (that is, layer 3 is a dense layer, which was not the case in LeNet-5 but I haven't been able to find a 100% faithful implementation using modern frameworks like Keras).

Nevertheless, I would expect it to exhibit some of the characteristics cited in the paper, namely these two metrics:

Significantly lower error (less than 1% cited in the paper, vs. ~4-5% for fully-connected case)
Tolerance of translational offsets in the images of up to 50% of the character height.

My results don't match this at all. See table below. LeNet-5 takes a 32x32 input image (formed by padding the 28x28 image), and I've applied various rotations (degrees) and translations (pixels). At a vertical shift of 5px, LeNet-5 fails pretty badly, contrary to the paper's claim that it can still identify characters.

Questions

What am I missing? Is the Keras implementation (which I did not write) flawed?
What are the advantages of LeNet-5 given these results?
What further experiments can I conduct to convince myself of its advantages?

Thank you :)


texts: How do I get Text Mining on Orange version 3.18

texts:
I just downloaded Orange version 3.18, but it is not displaying any of the Text Mining widgets (i.e. Corpus Viewer, Word count).
Is there a later version of Orange that I should download or is there a specific Add On I should download?
In Orange I have looked in "Options" and "Add-Ons" but there are no packages available to select.


texts: How do I plot between 2 variables in a summation equation?

texts:
I am new to Python and using Jupyter notebook.

Following is the summation equation that I have: $$y(t) = \sum_{i=1}^{t-1} a_{i}y(t-i)$$

How can I plot between variables y and t by formulating this summation equation?

$a_{i}$ is a constant but varies on every iteration. Any value of $a_{i}$ can be assumed in this case. Please give your opinions on this.


texts: Deploying multiple pre-trained model (tar.gz files) on Sagemaker in a single endpoint

texts:
We have followed the following steps:

Trained 5 TensorFlow models in local machine using 5 different training sets.
Saved those in .h5 format.
Converted those into tar.gz (Model1.tar.gz,...Model5.tar.gz) and uploaded it in the S3 bucket.
Successfully deployed a single model in an endpoint using the following code:

from sagemaker.tensorflow import TensorFlowModel
sagemaker_model = TensorFlowModel(model_data = tarS3Path + 'model{}.tar.gz'.format(1),
                                  role = role, framework_version='1.13',
                                  sagemaker_session = sagemaker_session)
predictor = sagemaker_model.deploy(initial_instance_count=1,
                                   instance_type='ml.m4.xlarge')
predictor.predict(data.values[:,0:])

The output was:
{'predictions': [[153.55], [79.8196], [45.2843]]}
Now the problem is that we cannot use 5 different deploy statements and create 5 different endpoints for 5 models. For this we followed two approaches:
i) Used MultiDataModal of Sagemaker
from sagemaker.multidatamodel import MultiDataModel
sagemaker_model1 = MultiDataModel(name = "laneMultiModels", model_data_prefix = tarS3Path,
                                 model=sagemaker_model, #This is the same sagemaker_model which is trained above
                                  #role = role, #framework_version='1.13',
                                  sagemaker_session = sagemaker_session)
predictor = sagemaker_model1.deploy(initial_instance_count=1,
                                   instance_type='ml.m4.xlarge')
predictor.predict(data.values[:,0:], target_model='model{}.tar.gz'.format(1))

Here we got an error at deploy stage which is as follows:
An error occurred (ValidationException) when calling the CreateModel operation: Your Ecr Image 763104351884.dkr.ecr.us-east-2.amazonaws.com/tensorflow-inference:1.13-cpu does not contain required com.amazonaws.sagemaker.capabilities.multi-models=true Docker label(s).
ii) Created endpoint manually
import boto3
import botocore
import sagemaker
sm_client = boto3.client('sagemaker')
image = sagemaker.image_uris.retrieve('knn','us-east-2')
container = {
    "Image": image,
    "ModelDataUrl": tarS3Path,
    "Mode": "MultiModel"
}
# Note if I replace "knn" by tensorflow it gives an error at this stage itself
response = sm_client.create_model(
              ModelName        = 'multiple-tar-models',
              ExecutionRoleArn = role,
              Containers       = [container])
response = sm_client.create_endpoint_config(
    EndpointConfigName = 'multiple-tar-models-endpointconfig',
    ProductionVariants=[{
        'InstanceType':        'ml.t2.medium',
        'InitialInstanceCount': 1,
        'InitialVariantWeight': 1,
        'ModelName':            'multiple-tar-models',
        'VariantName':          'AllTraffic'}])
response = sm_client.create_endpoint(
              EndpointName       = 'tarmodels-endpoint',
              EndpointConfigName = 'multiple-tar-models-endpointconfig')

Endpoint couldn't be created in this approach as well.


texts: Machine Learning & Image Recognition: How to start?

texts:
I've been a full stack web developer for 15 years now and would like to be involved in machine learning. There is already a specific scenario for this: We have a database with several million products and one product image each. There is also a database with about 5000 terms.

A product image is linked to several terms (usually 3 - 20), whereby the link still has a weighting (1-100%). The terms are always of a visual nature, that is, they describe a visually recognizable feature on the image.

The aim should now be to upload a new image (of course with thematic reference) and to get an answer with possible terms (including probability) based on the already classified images.

Do you have any advice on how best to start here? Is there a framework that comes close to this scenario? Is TensorFlow relevant for this task? What new language should I learn?

Thank you very much!


texts:
This is not a question as such but more likely to be verification (enhancement) of my current understanding. With the thought that it may help future visitors as well, I am taking liberty to make this post.

Question

Is the interpretation mentioned in attached image correct ?
As per my interpretation, Movie A doesn't seems to have much Action, nor Drama. What does that mean? Is it boring movie or what?
I would like to give some intuitive name to PC0 and PC1, what seems to be suitable name for it?


texts: Validation fraction parameter of sklearn's Gradient Boosted Classifier in use with time series data

texts:
I'm looking at using sklearn's Gradient Boosting Classifier (GBC) to predict the sign of stock returns. My question is regarding the parameter "validation_fraction" used for early stopping. Since I'm working with time series financial data, any bootstrapping/resampling etc where later observations are used as a training set to predict past observations would be problematic. For example when I use the RandomForestClassifier model I have to set bootstrap to False.
My question is, with this parameter, does the GBC use any resample and/or bootstrapping of any kind ? If so I would have to find a way to modify the source code to ensure the integrity of the training/validation set split.
Thanks.


texts: Binary classifier using Keras with backend Tensorflow with a Binary output

texts:
I am trying to build a binary classifier with tensorflow.keras
Currently unable to identify a solution to having the model generating only 0s and 1s.

The code for compiling my tensorflow model.

from __future__ import absolute_import, division, print_function, unicode_literals
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

def build_model(train_dataset):
  model = keras.Sequential([
    layers.Dense(10, activation='relu', input_shape=[len(train_dataset.keys())]),
    layers.Dense(1, activation='sigmoid')
  ])
  optimizer = 'adam'

  model.compile(loss='binary_crossentropy',
                optimizer=optimizer,
                metrics=[tf.keras.metrics.Precision(), tf.keras.metrics.Recall(), tf.keras.metrics.Accuracy()])
  return model

model = build_model(X)

After calling model.fit(X,y) and model.predict(X_test) an array of numbers are produced as prediction values:

array([[8.3726919e-01],
       [9.1233850e-04],
       [8.3726919e-01],
       ...,
       [4.6819448e-05],
       [1.5565268e-08],
       [0.0000000e+00]], dtype=float32)

My current solution to the problem is to get the output as a number in the range (0,1) from the sigmoid activation function; and then transforming the values to 0 or 1

reset_predictions = []
for p in predictions:
    if p >= 0.5:
        reset_predictions.append(1)
    else:
        reset_predictions.append(0)

However this is not ideal; what I hope to achieve is for the model to classify 0 or 1 for me without an external function.


texts: How to select features based on feature importance using SelectFromModel?

texts:
I would appreciate if you could let me know how to select features based on feature importance using SelectFromModel.

I wrote:

# data
X = np.array(pd.read_csv('who_X_1.csv',header=None).values)
y = np.array(pd.read_csv('who_Y_1.csv',header=None).values.ravel())
indices = np.arange(y.shape[0])

# # Divide Data into Train and Test
X_train, X_test, y_train, y_test,idx_train,idx_test = train_test_split(X, yy,indices,stratify=yy,test_size=0.3, random_state=42)

scaler = StandardScaler()

# # Compute Cohen's Kappa or Auc as scoring criterion due to imbalanced data set
kappa_scorer = make_scorer(cohen_kappa_score)
auc_scorer=make_scorer(roc_auc_score)
F_measure_scorer = make_scorer(f1_score)

##hyperparameter

param_grid = {
    'clf__colsample_bytree': [i/10.0 for i in range(7,10)],
    #"clf__subsample"  : [i/10.0 for i in range(5,10)],
    #'clf__max_depth':range(5,15,1),
    #'clf__gamma':[i/10.0 for i in range(0,5)],
    #'clf__reg_alpha':[1e-5, 1e-2, 0.1, 1, 100]

            }

##Classifier
xg=XGBClassifier(max_depth=3,
                 learning_rate=0.05,
                 n_estimators=350,
                 objective="binary:logistic",
                 booster="gbtree",
                 gamma=0,
                 min_child_weight=0.8,
                 subsample=1,
                 colsample_bylevel=1,
                 colsample_bytree=0.6,
                 reg_alpha=0.001,
                 reg_lambda=1,
                 scale_pos_weight=22,
                 random_state=4,n_jobs=-1)
pipe=Pipeline(steps=[('pre',scaler),
                    ('clf',xg)])

rg_cv = GridSearchCV(pipe, param_grid, cv=5, scoring = 'f1')
rg_cv.fit(X_train, y_train)
print("Tuned rf best params: {}".format(rg_cv.best_params_))

# Use SelectFromModel
thresholds = np.sort(rg_cv.best_estimator_.named_steps["clf"].feature_importances_)
for thresh in thresholds:
    # select features using threshold
    selection = SelectFromModel(rg_cv, threshold=thresh, prefit=True)
    select_X_train = selection.transform(X_train)

    # train model
    selection_model = rg_cv
    selection_model.fit(select_X_train, y_train)

    # eval model
    select_X_test = selection.transform(X_test)
    y_pred = selection_model.predict(select_X_test)
    predictions = [round(value) for value in y_pred]
    accuracy = accuracy_score(y_test, predictions)
    print("Thresh=%.3f, n=%d, Accuracy: %.2f%%" % (thresh, select_X_train.shape[1], accuracy * 100.0))
    print(confusion_matrix(y_test, predictions))
    print(classification_report(y_test, predictions))

However, the following error occurred:

ValueError: The underlying estimator GridSearchCV has no `coef_` or `feature_importances_` attribute. Either pass a fitted estimator to SelectFromModel or call fit before calling transform.


texts: Python: Detect if data of a time series stays constant, increases or decreases

texts:
i need to analyse and later try to improve (integrate a filter) for measurement data that i compare to accurate reference data with python.

First i want to calculate the mean offset and the standard deviation of the measurement data overall, during halt, increasing and decreasing state.

How can i automatically detect and mark sections in which the reference system data stays constant, increases or decreases and use this information later for the statistical analysis.

I already tried to write a simple algorithm myself but the problem is though the reference data is given precise enough so you can calculate the difference of two neighbouring data points and it is zero if no change takes place this case is also in the majority during sections of change (increase, decrease). So the algorithm can't really mark the state of the data points reliable.

I already researched a little bit and here are some keywords that seemed interesting for my problem: time series analysis, anomaly detection, novelty detection, change point detection, structural changes, One-Class Support Vector Machines

So my question would be if sb. could give me a pointer towards the right direction (correct method, python package, tutorial, example) so i can solve this problem myself.


texts: GPU shows 0 utilization even when tensors and model are mounted on the gpu?

texts:
I am trying to run some PyTorch scripts on a remote GPU server. While calling the script in the ubuntu terminal i start as:CUDA_VISIBLE_DEVICES=0(or whichever is available) python3 <script.py>. Also, used the following snippet in the code and used .to(device) on the model, input and target tensors.
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)

I have confirmed that my model and data and target tensors are mounted on the cuda device. But the GPU shows 0 percentage utilization all through the run. What could I be missing?


texts: How to deal with analyzing optional survey data

texts:
Consider this scenario. There is a website that ranks every type of sandwich. For each type of sandwich, users fill out a survey about the following: A) how tasty it is (on a scale from 1-5); B) how filling it is (on a scale from 1 to 5); C*) if it is spicy (-1 or 1).

The website displays the average value of the answers to Question A, the average value of the answers to Question B, and the sum of the answers to Question C.

Question A and Question B are mandatory. Question C, however, is optional. I have the number of user-submitted ratings for each sandwich, and I have the sum of the answers to C.

I want to compare the spiciness of different sandwiches. Since different sandwiches have different rankings, I cannot simply compare the sum of the answers to Question C. I also cannot weight it as "Sum of Answers to Question C/Number of Ratings" since Question C is optional and some people won't answer it.

What would be the best way to go about comparing the spiciness of different sandwiches under these constraints?


texts: Is knn similar to this version of k-means?

texts:
If we use k-means in a dataset where k is equal to the number of points in the dataset, and each cluster is made out of only a point. Considering that we have given a distance method, we can classify a test instance associating it with the cluster whose centroid is nearest to that given point. The centroids are not changing in the test phase.
With what automatic classification method is the above strategy similar to?
Yes, this is a homework. My answer was knn, is this correct? But I do not know how to argue this properly.


texts: How to schedule importing data files from SFTP server located on compute engine instance into BigQuery?

texts:
What I want to achieve:
Transfer hourly coming data files onto a SFTP file server located on a compute engine VM  from several different feeds into Bigquery with real-time updates effectively & cost-efficiently.
Context:

The software I am trying to import data from is an old legacy software and does not support direct exports to cloud. So direct connection from software to cloud isn't an option.

It does however support exporting data to a SFTP server. Which is not available via any GCP tools directly.

So I have setup a manual SFTP server using vsftpd on a compute engine VM instance with expandable storage then giving it a static IP and hardwired that IP into my software. Data now comes to the compute engine instance on hourly intervals seamlessly.

Files are generated on hourly basis. Thus a different file for each hour. However they might contain some duplication .i.e. some of the end records of previous hour's file may overlap with the beginning of the current hour's file.

Files are coming from different source feeds and I have feed names in the filenames so ever-growing data on my compute engine VM instance look like:
feed1_210301_0500.csv
feed2_210301_0500.csv
feed3_210301_0500.csv
feed1_210301_0600.csv
feed2_210301_0600.csv
feed3_210301_0600.csv
feed1_210301_0700.csv
feed2_210301_0700.csv
feed3_210301_0700.csv
...

What I have tried:

I have set Bigquery access & cloud storage permissions within VM instance to access data from VM instance onto BigQuery:

I have tried importing data into BigQuery directly as well as on google cloud storage to import data from there and yet there is no option to directly import data from VM instance to BigQuery nor I can somehow import data from VM to GCS then to load into BigQuery but there is no option for that and documentation is silent on the matter of scheduled transfers as well.

There are some external data transfer services like Fivetran and HevoData but they are relatively expensive and also seem much of a overkill as both my source or destination is on the GCP and it wont be much different than having a third VM and scheduling some scripts for imports. (Which BTW is my current workaround :D i.e. Using python scripts to stream data into BigQuery as explained here)

Currently I am exploring DataFusion which is only free for 120 hrs each month, has extra costs for underlying Dataprep pipelines and not sure if its the right way to go about. Also I am currently exploring tools like Cloud Scheduler & Cloud Composer to see if any fits my data needs but as of now could not find a viable solution.

I am happy to learn any new tools and technologies and any advice bettering the situation in anyway is also appreciated


texts: after grouping to minimum value in pandas, how to display the matching row result entirely along min() value